Sep 18 13:13:01.393779 ip-10-0-131-221 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 18 13:13:01.393793 ip-10-0-131-221 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 18 13:13:01.393802 ip-10-0-131-221 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 18 13:13:01.394154 ip-10-0-131-221 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 18 13:13:11.592385 ip-10-0-131-221 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 18 13:13:11.592405 ip-10-0-131-221 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot bea14e5c25e44799afe3436cbdfaf8f8 --
Sep 18 13:15:01.083230 ip-10-0-131-221 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 13:15:04.671851 ip-10-0-131-221 kubenswrapper[2586]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 13:15:04.671851 ip-10-0-131-221 kubenswrapper[2586]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 13:15:04.671851 ip-10-0-131-221 kubenswrapper[2586]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 13:15:04.671851 ip-10-0-131-221 kubenswrapper[2586]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 13:15:04.671851 ip-10-0-131-221 kubenswrapper[2586]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 13:15:04.672546 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.671933    2586 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 13:15:04.675389 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675372    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:15:04.675389 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675388    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:15:04.675453 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675392    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:15:04.675453 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675396    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:15:04.675453 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675400    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:15:04.675453 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675403    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:15:04.675453 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675406    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:15:04.675453 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675409    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:15:04.675453 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675412    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:15:04.675453 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675415    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:15:04.675453 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675418    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:15:04.675453 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675421    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:15:04.675453 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675424    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:15:04.675453 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675427    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:15:04.675453 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675429    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:15:04.675453 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675432    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:15:04.675453 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675434    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:15:04.675453 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675437    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:15:04.675453 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675440    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:15:04.675453 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675449    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:15:04.675453 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675452    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:15:04.675453 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675455    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:15:04.675943 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675458    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:15:04.675943 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675462    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:15:04.675943 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675466    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:15:04.675943 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675469    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:15:04.675943 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675472    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:15:04.675943 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675475    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:15:04.675943 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675477    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:15:04.675943 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675481    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:15:04.675943 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675483    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:15:04.675943 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675486    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:15:04.675943 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675489    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:15:04.675943 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675492    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:15:04.675943 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675496    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:15:04.675943 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675498    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:15:04.675943 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675501    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:15:04.675943 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675503    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:15:04.675943 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675506    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:15:04.675943 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675508    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:15:04.675943 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675511    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:15:04.676442 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675513    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:15:04.676442 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675516    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:15:04.676442 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675519    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:15:04.676442 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675522    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:15:04.676442 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675524    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:15:04.676442 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675527    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:15:04.676442 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675530    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:15:04.676442 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675532    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:15:04.676442 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675536    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:15:04.676442 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675538    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:15:04.676442 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675540    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:15:04.676442 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675543    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:15:04.676442 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675546    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:15:04.676442 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675548    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:15:04.676442 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675551    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:15:04.676442 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675554    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:15:04.676442 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675557    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:15:04.676442 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675559    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:15:04.676442 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675562    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:15:04.676442 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675565    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:15:04.676928 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675567    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:15:04.676928 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675570    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:15:04.676928 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675572    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:15:04.676928 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675575    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:15:04.676928 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675578    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:15:04.676928 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675580    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:15:04.676928 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675583    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:15:04.676928 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675585    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:15:04.676928 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675588    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:15:04.676928 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675590    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:15:04.676928 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675593    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:15:04.676928 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675595    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:15:04.676928 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675598    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:15:04.676928 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675600    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:15:04.676928 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675603    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:15:04.676928 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675606    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:15:04.676928 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675608    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:15:04.676928 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675611    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:15:04.676928 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675616    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:15:04.676928 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675618    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:15:04.677417 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675621    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:15:04.677417 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675624    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:15:04.677417 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675627    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:15:04.677417 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675629    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:15:04.677417 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675632    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:15:04.677417 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675636    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:15:04.677417 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675640    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:15:04.677417 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675643    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:15:04.677417 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675646    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:15:04.677417 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675649    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:15:04.677417 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675652    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:15:04.677417 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675655    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:15:04.677417 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675658    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:15:04.677417 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675660    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:15:04.677417 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675663    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:15:04.677417 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675666    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:15:04.677417 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675669    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:15:04.677417 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.675672    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:15:04.677417 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676385    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:15:04.677417 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676391    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:15:04.677923 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676395    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:15:04.677923 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676398    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:15:04.677923 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676401    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:15:04.677923 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676403    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:15:04.677923 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676406    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:15:04.677923 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676409    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:15:04.677923 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676412    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:15:04.677923 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676414    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:15:04.677923 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676417    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:15:04.677923 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676420    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:15:04.677923 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676423    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:15:04.677923 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676425    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:15:04.677923 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676428    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:15:04.677923 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676430    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:15:04.677923 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676433    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:15:04.677923 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676435    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:15:04.677923 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676438    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:15:04.677923 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676441    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:15:04.677923 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676446    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:15:04.677923 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676448    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:15:04.678424 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676451    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:15:04.678424 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676455    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:15:04.678424 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676459    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:15:04.678424 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676462    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:15:04.678424 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676465    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:15:04.678424 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676468    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:15:04.678424 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676471    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:15:04.678424 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676475    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:15:04.678424 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676479    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:15:04.678424 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676481    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:15:04.678424 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676484    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:15:04.678424 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676487    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:15:04.678424 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676490    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:15:04.678424 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676492    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:15:04.678424 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676495    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:15:04.678424 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676498    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:15:04.678424 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676500    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:15:04.678424 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676503    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:15:04.678424 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676506    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:15:04.678882 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676508    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:15:04.678882 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676511    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:15:04.678882 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676514    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:15:04.678882 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676517    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:15:04.678882 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676519    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:15:04.678882 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676522    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:15:04.678882 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676525    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:15:04.678882 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676528    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:15:04.678882 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676531    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:15:04.678882 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676533    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:15:04.678882 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676535    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:15:04.678882 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676539    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:15:04.678882 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676541    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:15:04.678882 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676544    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:15:04.678882 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676546    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:15:04.678882 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676549    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:15:04.678882 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676551    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:15:04.678882 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676554    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:15:04.678882 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676556    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:15:04.678882 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676559    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:15:04.679379 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676561    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:15:04.679379 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676564    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:15:04.679379 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676566    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:15:04.679379 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676569    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:15:04.679379 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676571    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:15:04.679379 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676574    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:15:04.679379 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676576    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:15:04.679379 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676579    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:15:04.679379 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676581    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:15:04.679379 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676585    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:15:04.679379 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676589    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:15:04.679379 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676592    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:15:04.679379 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676596    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:15:04.679379 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676599    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:15:04.679379 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676601    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:15:04.679379 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676605    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:15:04.679379 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676608    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:15:04.679379 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676610    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:15:04.679379 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676613    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:15:04.679379 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676616    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:15:04.679855 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676618    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:15:04.679855 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676621    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:15:04.679855 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676624    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:15:04.679855 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676627    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:15:04.679855 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676629    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:15:04.679855 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676632    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:15:04.679855 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676635    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:15:04.679855 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676637    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:15:04.679855 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676640    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:15:04.679855 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676642    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:15:04.679855 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676645    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:15:04.679855 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676647    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:15:04.679855 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676650    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:15:04.679855 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676652    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:15:04.679855 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676655    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:15:04.679855 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676657    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:15:04.679855 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676660    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:15:04.679855 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.676663    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:15:04.679855 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677049    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:15:04.679855 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677055    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:15:04.680356 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677058    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:15:04.680356 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677061    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:15:04.680356 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677063    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:15:04.680356 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677066    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:15:04.680356 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677069    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:15:04.680356 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677072    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:15:04.680356 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677074    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:15:04.680356 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677078    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:15:04.680356 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677081    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:15:04.680356 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677083    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:15:04.680356 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677087    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:15:04.680356 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677089    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:15:04.680356 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677092    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:15:04.680356 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677095    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:15:04.680356 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677097    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:15:04.680356 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677101    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:15:04.680356 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677104    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:15:04.680356 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677106    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:15:04.680356 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677109    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:15:04.680356 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677111    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:15:04.680877 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677114    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:15:04.680877 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677117    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:15:04.680877 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677120    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:15:04.680877 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677122    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:15:04.680877 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677125    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:15:04.680877 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677127    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:15:04.680877 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677130    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:15:04.680877 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677133    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:15:04.680877 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677135    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:15:04.680877 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677138    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:15:04.680877 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677140    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:15:04.680877 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677143    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:15:04.680877 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677145    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:15:04.680877 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677148    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:15:04.680877 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677151    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:15:04.680877 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677153    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:15:04.680877 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677155    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:15:04.680877 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677158    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:15:04.680877 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677161    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:15:04.680877 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677164    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:15:04.681395 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677167    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:15:04.681395 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677170    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:15:04.681395 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677173    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:15:04.681395 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677175    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:15:04.681395 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677178    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:15:04.681395 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677180    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:15:04.681395 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677183    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:15:04.681395 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677187    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:15:04.681395 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677189    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:15:04.681395 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677192    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:15:04.681395 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677195    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:15:04.681395 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677197    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:15:04.681395 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677199    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:15:04.681395 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677202    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:15:04.681395 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677205    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:15:04.681395 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677208    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:15:04.681395 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677210    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:15:04.681395 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677213    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:15:04.681395 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677215    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:15:04.681395 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677218    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:15:04.681890 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677220    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:15:04.681890 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677223    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:15:04.681890 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677225    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:15:04.681890 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677228    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:15:04.681890 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677231    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:15:04.681890 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677233    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:15:04.681890 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677236    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:15:04.681890 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677238    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:15:04.681890 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677241    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:15:04.681890 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677243    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:15:04.681890 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677246    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:15:04.681890 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677249    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:15:04.681890 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677251    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:15:04.681890 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677254    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:15:04.681890 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677257    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:15:04.681890 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677259    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:15:04.681890 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677262    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:15:04.681890 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677264    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:15:04.681890 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677267    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:15:04.681890 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677270    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:15:04.681890 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677272    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:15:04.682414 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677276    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:15:04.682414 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677280    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:15:04.682414 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677284    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:15:04.682414 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677288    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:15:04.682414 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677291    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:15:04.682414 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677294    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:15:04.682414 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677297    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:15:04.682414 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677300    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:15:04.682414 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677302    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:15:04.682414 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677305    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:15:04.682414 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677308    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:15:04.682414 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677311    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:15:04.682414 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677314    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:15:04.682414 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677317    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:15:04.682414 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677320    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:15:04.682414 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.677322    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:15:04.682414 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677406    2586 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 13:15:04.682414 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677449    2586 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 13:15:04.682414 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677457    2586 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 13:15:04.682868 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677465    2586 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 13:15:04.682868 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677469    2586 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 13:15:04.682868 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677472    2586 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 13:15:04.682868 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677476    2586 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 13:15:04.682868 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677481    2586 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 13:15:04.682868 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677484    2586 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 13:15:04.682868 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677488    2586 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 13:15:04.682868 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677491    2586 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 13:15:04.682868 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677494    2586 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 13:15:04.682868 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677497    2586 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 13:15:04.682868 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677500    2586 flags.go:64] FLAG: --cgroup-root=""
Sep 18 13:15:04.682868 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677504    2586 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 13:15:04.682868 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677507    2586 flags.go:64] FLAG: --client-ca-file=""
Sep 18 13:15:04.682868 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677509    2586 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 13:15:04.682868 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677512    2586 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 13:15:04.682868 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677516    2586 flags.go:64] FLAG: --cluster-domain=""
Sep 18 13:15:04.682868 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677519    2586 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 13:15:04.682868 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677522    2586 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 13:15:04.682868 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677525    2586 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 13:15:04.682868 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677529    2586 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 13:15:04.682868 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677533    2586 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 13:15:04.682868 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677540    2586 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 13:15:04.682868 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677543    2586 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 13:15:04.683448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677547    2586 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 13:15:04.683448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677550    2586 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 13:15:04.683448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677553    2586 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 13:15:04.683448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677556    2586 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 13:15:04.683448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677559    2586 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 13:15:04.683448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677562    2586 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 13:15:04.683448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677570    2586 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 13:15:04.683448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677572    2586 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 13:15:04.683448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677575    2586 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 13:15:04.683448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677580    2586 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 13:15:04.683448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677583    2586 flags.go:64] FLAG: --enable-server="true"
Sep 18 13:15:04.683448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677586    2586 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 13:15:04.683448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677592    2586 flags.go:64] FLAG: --event-burst="100"
Sep 18 13:15:04.683448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677595    2586 flags.go:64] FLAG: --event-qps="50"
Sep 18 13:15:04.683448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677598    2586 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 13:15:04.683448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677601    2586 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 13:15:04.683448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677604    2586 flags.go:64] FLAG: --eviction-hard=""
Sep 18 13:15:04.683448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677608    2586 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 13:15:04.683448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677611    2586 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 13:15:04.683448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677614    2586 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 13:15:04.683448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677617    2586 flags.go:64] FLAG: --eviction-soft=""
Sep 18 13:15:04.683448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677620    2586 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 13:15:04.683448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677623    2586 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 13:15:04.683448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677625    2586 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 13:15:04.683448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677628    2586 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 13:15:04.684150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677631    2586 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 13:15:04.684150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677634    2586 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 13:15:04.684150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677637    2586 flags.go:64] FLAG: --feature-gates=""
Sep 18 13:15:04.684150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677641    2586 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 13:15:04.684150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677644    2586 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 13:15:04.684150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677647    2586 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 13:15:04.684150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677650    2586 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 13:15:04.684150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677653    2586 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 13:15:04.684150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677656    2586 flags.go:64] FLAG: --help="false"
Sep 18 13:15:04.684150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677659    2586 flags.go:64] FLAG: --hostname-override="ip-10-0-131-221.ec2.internal"
Sep 18 13:15:04.684150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677662    2586 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 13:15:04.684150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677665    2586 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 13:15:04.684150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677668    2586 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 13:15:04.684150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677671    2586 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 13:15:04.684150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677675    2586 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 13:15:04.684150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677677    2586 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 13:15:04.684150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677682    2586 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 13:15:04.684150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677685    2586 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 13:15:04.684150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677688    2586 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 13:15:04.684150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677691    2586 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 13:15:04.684150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677694    2586 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 13:15:04.684150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677697    2586 flags.go:64] FLAG: --kube-reserved=""
Sep 18 13:15:04.684150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677700    2586 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 13:15:04.684150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677703    2586 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 13:15:04.684819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677706    2586 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 13:15:04.684819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677709    2586 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 13:15:04.684819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677712    2586 flags.go:64] FLAG: --lock-file=""
Sep 18 13:15:04.684819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677715    2586 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 13:15:04.684819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677718    2586 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 13:15:04.684819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677721    2586 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 13:15:04.684819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677726    2586 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 13:15:04.684819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677729    2586 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 13:15:04.684819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677731    2586 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 13:15:04.684819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677734    2586 flags.go:64] FLAG: --logging-format="text"
Sep 18 13:15:04.684819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677737    2586 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 13:15:04.684819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677740    2586 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 13:15:04.684819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677744    2586 flags.go:64] FLAG: --manifest-url=""
Sep 18 13:15:04.684819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677746    2586 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 13:15:04.684819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677751    2586 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 13:15:04.684819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677753    2586 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 13:15:04.684819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677757    2586 flags.go:64] FLAG: --max-pods="110"
Sep 18 13:15:04.684819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677760    2586 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 13:15:04.684819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677763    2586 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 13:15:04.684819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677766    2586 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 13:15:04.684819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677769    2586 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 13:15:04.684819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677772    2586 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 13:15:04.684819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677775    2586 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 13:15:04.684819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677778    2586 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 13:15:04.684819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677786    2586 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 13:15:04.685924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677789    2586 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 13:15:04.685924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677791    2586 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 13:15:04.685924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677794    2586 flags.go:64] FLAG: --pod-cidr=""
Sep 18 13:15:04.685924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677797    2586 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 13:15:04.685924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677802    2586 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 13:15:04.685924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677805    2586 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 13:15:04.685924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677808    2586 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 13:15:04.685924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677811    2586 flags.go:64] FLAG: --port="10250"
Sep 18 13:15:04.685924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677815    2586 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 13:15:04.685924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677818    2586 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01abeed94ad6a6b68"
Sep 18 13:15:04.685924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677821    2586 flags.go:64] FLAG: --qos-reserved=""
Sep 18 13:15:04.685924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677824    2586 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 13:15:04.685924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677827    2586 flags.go:64] FLAG: --register-node="true"
Sep 18 13:15:04.685924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677830    2586 flags.go:64] FLAG: --register-with-taints=""
Sep 18 13:15:04.685924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677833    2586 flags.go:64] FLAG: --registry-burst="10"
Sep 18 13:15:04.685924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677836    2586 flags.go:64] FLAG: --registry-qps="5"
Sep 18 13:15:04.685924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677839    2586 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 13:15:04.685924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677842    2586 flags.go:64] FLAG: --reserved-memory=""
Sep 18 13:15:04.685924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677845    2586 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 13:15:04.685924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677849    2586 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 13:15:04.685924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677852    2586 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 13:15:04.685924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677855    2586 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 13:15:04.685924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677857    2586 flags.go:64] FLAG: --runonce="false"
Sep 18 13:15:04.685924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677860    2586 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 13:15:04.685924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677863    2586 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 13:15:04.686531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677866    2586 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 13:15:04.686531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677869    2586 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 13:15:04.686531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677872    2586 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 13:15:04.686531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677875    2586 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 13:15:04.686531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677878    2586 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 13:15:04.686531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677881    2586 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 13:15:04.686531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677884    2586 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 13:15:04.686531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677888    2586 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 13:15:04.686531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677891    2586 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 13:15:04.686531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677893    2586 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 13:15:04.686531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677896    2586 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 13:15:04.686531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677913    2586 flags.go:64] FLAG: --system-cgroups=""
Sep 18 13:15:04.686531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677916    2586 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 13:15:04.686531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677922    2586 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 13:15:04.686531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677924    2586 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 13:15:04.686531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677927    2586 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 13:15:04.686531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677932    2586 flags.go:64] FLAG: --tls-min-version=""
Sep 18 13:15:04.686531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677935    2586 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 13:15:04.686531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677938    2586 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 13:15:04.686531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677941    2586 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 13:15:04.686531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677944    2586 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 13:15:04.686531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677947    2586 flags.go:64] FLAG: --v="2"
Sep 18 13:15:04.686531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677951    2586 flags.go:64] FLAG: --version="false"
Sep 18 13:15:04.686531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677955    2586 flags.go:64] FLAG: --vmodule=""
Sep 18 13:15:04.686531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677961    2586 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 13:15:04.687155 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.677964    2586 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 13:15:04.687155 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678091    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:15:04.687155 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678095    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:15:04.687155 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678097    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:15:04.687155 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678101    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:15:04.687155 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678103    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:15:04.687155 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678106    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:15:04.687155 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678109    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:15:04.687155 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678112    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:15:04.687155 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678114    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:15:04.687155 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678117    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:15:04.687155 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678120    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:15:04.687155 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678123    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:15:04.687155 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678125    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:15:04.687155 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678128    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:15:04.687155 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678131    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:15:04.687155 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678134    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:15:04.687155 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678136    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:15:04.687155 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678139    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:15:04.687155 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678141    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:15:04.687650 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678144    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:15:04.687650 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678147    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:15:04.687650 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678149    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:15:04.687650 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678152    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:15:04.687650 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678155    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:15:04.687650 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678158    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:15:04.687650 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678160    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:15:04.687650 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678163    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:15:04.687650 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678166    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:15:04.687650 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678168    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:15:04.687650 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678171    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:15:04.687650 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678175    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:15:04.687650 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678179    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:15:04.687650 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678183    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:15:04.687650 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678186    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:15:04.687650 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678189    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:15:04.687650 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678192    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:15:04.687650 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678195    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:15:04.687650 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678198    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:15:04.688153 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678201    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:15:04.688153 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678204    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:15:04.688153 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678206    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:15:04.688153 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678209    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:15:04.688153 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678211    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:15:04.688153 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678214    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:15:04.688153 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678217    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:15:04.688153 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678220    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:15:04.688153 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678223    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:15:04.688153 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678226    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:15:04.688153 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678228    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:15:04.688153 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678231    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:15:04.688153 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678234    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:15:04.688153 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678236    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:15:04.688153 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678239    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:15:04.688153 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678242    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:15:04.688153 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678245    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:15:04.688153 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678247    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:15:04.688153 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678250    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:15:04.688153 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678253    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:15:04.688668 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678256    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:15:04.688668 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678258    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:15:04.688668 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678261    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:15:04.688668 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678264    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:15:04.688668 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678269    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:15:04.688668 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678272    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:15:04.688668 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678274    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:15:04.688668 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678278    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:15:04.688668 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678282    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:15:04.688668 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678285    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:15:04.688668 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678288    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:15:04.688668 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678291    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:15:04.688668 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678294    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:15:04.688668 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678297    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:15:04.688668 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678300    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:15:04.688668 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678302    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:15:04.688668 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678305    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:15:04.688668 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678308    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:15:04.688668 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678311    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:15:04.688668 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678314    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:15:04.689197 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678318    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:15:04.689197 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678321    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:15:04.689197 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678324    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:15:04.689197 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678327    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:15:04.689197 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678330    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:15:04.689197 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678335    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:15:04.689197 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678338    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:15:04.689197 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678341    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:15:04.689197 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678344    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:15:04.689197 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678346    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:15:04.689197 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678349    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:15:04.689197 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678352    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:15:04.689197 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678355    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:15:04.689197 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678357    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:15:04.689197 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678360    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:15:04.689197 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678362    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:15:04.689197 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678367    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:15:04.689197 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678369    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:15:04.689197 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678372    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:15:04.689197 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678374    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:15:04.689704 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.678377    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:15:04.689704 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.678383    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 13:15:04.689704 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.686353    2586 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 13:15:04.689704 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.686368    2586 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 13:15:04.689704 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.686389    2586 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 13:15:04.689704 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.686395    2586 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 13:15:04.689704 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686451    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:15:04.689704 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686456    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:15:04.689704 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686460    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:15:04.689704 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686463    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:15:04.689704 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686466    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:15:04.689704 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686469    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:15:04.689704 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686472    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:15:04.689704 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686474    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:15:04.689704 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686477    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:15:04.690113 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686480    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:15:04.690113 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686483    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:15:04.690113 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686486    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:15:04.690113 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686488    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:15:04.690113 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686491    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:15:04.690113 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686493    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:15:04.690113 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686496    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:15:04.690113 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686499    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:15:04.690113 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686502    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:15:04.690113 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686504    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:15:04.690113 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686508    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:15:04.690113 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686510    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:15:04.690113 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686513    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:15:04.690113 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686516    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:15:04.690113 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686519    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:15:04.690113 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686530    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:15:04.690113 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686533    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:15:04.690113 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686536    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:15:04.690113 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686539    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:15:04.690568 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686542    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:15:04.690568 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686545    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:15:04.690568 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686548    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:15:04.690568 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686550    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:15:04.690568 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686553    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:15:04.690568 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686557    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:15:04.690568 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686562    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:15:04.690568 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686565    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:15:04.690568 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686569    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:15:04.690568 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686572    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:15:04.690568 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686574    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:15:04.690568 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686577    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:15:04.690568 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686580    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:15:04.690568 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686583    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:15:04.690568 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686586    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:15:04.690568 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686588    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:15:04.690568 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686591    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:15:04.690568 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686593    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:15:04.690568 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686597    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:15:04.691059 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686601    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:15:04.691059 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686604    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:15:04.691059 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686607    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:15:04.691059 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686609    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:15:04.691059 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686612    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:15:04.691059 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686615    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:15:04.691059 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686617    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:15:04.691059 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686620    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:15:04.691059 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686623    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:15:04.691059 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686626    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:15:04.691059 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686629    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:15:04.691059 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686631    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:15:04.691059 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686634    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:15:04.691059 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686637    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:15:04.691059 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686640    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:15:04.691059 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686643    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:15:04.691059 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686646    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:15:04.691059 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686648    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:15:04.691059 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686650    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:15:04.691059 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686653    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:15:04.691544 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686655    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:15:04.691544 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686658    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:15:04.691544 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686661    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:15:04.691544 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686663    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:15:04.691544 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686666    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:15:04.691544 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686668    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:15:04.691544 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686671    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:15:04.691544 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686674    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:15:04.691544 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686677    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:15:04.691544 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686679    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:15:04.691544 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686681    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:15:04.691544 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686684    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:15:04.691544 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686687    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:15:04.691544 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686690    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:15:04.691544 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686692    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:15:04.691544 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686695    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:15:04.691544 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686697    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:15:04.691544 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686700    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:15:04.691544 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686702    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:15:04.691544 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686705    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:15:04.692095 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686708    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:15:04.692095 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686711    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:15:04.692095 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686714    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:15:04.692095 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686716    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:15:04.692095 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686720    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:15:04.692095 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686723    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:15:04.692095 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686726    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:15:04.692095 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686729    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:15:04.692095 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686731    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:15:04.692095 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686734    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:15:04.692095 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686736    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:15:04.692095 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686739    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:15:04.692095 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.686744    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 13:15:04.692095 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686857    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 13:15:04.692095 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686862    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 13:15:04.692485 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686865    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 13:15:04.692485 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686867    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 13:15:04.692485 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686870    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 13:15:04.692485 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686873    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 13:15:04.692485 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686875    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 13:15:04.692485 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686878    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 13:15:04.692485 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686881    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 13:15:04.692485 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686884    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 13:15:04.692485 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686886    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 13:15:04.692485 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686889    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 13:15:04.692485 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686892    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 13:15:04.692485 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686895    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 13:15:04.692485 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686898    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 13:15:04.692485 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686918    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 13:15:04.692485 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686921    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 13:15:04.692485 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686926    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 13:15:04.692485 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686930    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 13:15:04.692485 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686933    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 13:15:04.692485 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686936    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 13:15:04.692485 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686939    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 13:15:04.692978 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686942    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 13:15:04.692978 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686945    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 13:15:04.692978 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686948    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 13:15:04.692978 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686951    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 13:15:04.692978 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686953    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 13:15:04.692978 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686956    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 13:15:04.692978 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686958    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 13:15:04.692978 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686961    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 13:15:04.692978 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686964    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 13:15:04.692978 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686967    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 13:15:04.692978 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686969    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 13:15:04.692978 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686972    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 13:15:04.692978 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686975    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 13:15:04.692978 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686977    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 13:15:04.692978 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686980    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 13:15:04.692978 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686983    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 13:15:04.692978 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686985    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 13:15:04.692978 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686988    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 13:15:04.692978 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686991    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 13:15:04.692978 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686993    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 13:15:04.693447 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686996    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 13:15:04.693447 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.686998    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 13:15:04.693447 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687001    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 13:15:04.693447 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687005    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 13:15:04.693447 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687008    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 13:15:04.693447 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687011    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 13:15:04.693447 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687013    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 13:15:04.693447 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687016    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 13:15:04.693447 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687019    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 13:15:04.693447 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687021    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 13:15:04.693447 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687024    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 13:15:04.693447 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687027    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 13:15:04.693447 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687029    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 13:15:04.693447 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687032    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 13:15:04.693447 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687034    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 13:15:04.693447 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687037    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 13:15:04.693447 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687039    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 13:15:04.693447 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687042    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 13:15:04.693447 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687045    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 13:15:04.693920 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687047    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 13:15:04.693920 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687050    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 13:15:04.693920 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687052    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 13:15:04.693920 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687055    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 13:15:04.693920 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687057    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 13:15:04.693920 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687061    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 13:15:04.693920 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687064    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 13:15:04.693920 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687067    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 13:15:04.693920 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687070    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 13:15:04.693920 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687073    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 13:15:04.693920 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687075    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 13:15:04.693920 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687078    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 13:15:04.693920 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687080    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 13:15:04.693920 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687083    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 13:15:04.693920 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687086    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 13:15:04.693920 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687088    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 13:15:04.693920 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687091    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 13:15:04.693920 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687095    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 13:15:04.693920 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687097    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 13:15:04.693920 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687100    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 13:15:04.694472 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687103    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 13:15:04.694472 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687106    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 13:15:04.694472 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687108    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 13:15:04.694472 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687111    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 13:15:04.694472 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687114    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 13:15:04.694472 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687116    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 13:15:04.694472 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687119    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 13:15:04.694472 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687123    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 13:15:04.694472 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687126    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 13:15:04.694472 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687129    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 13:15:04.694472 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687131    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 13:15:04.694472 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687134    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 13:15:04.694472 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687136    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 13:15:04.694472 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687139    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 13:15:04.694472 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687141    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 13:15:04.694472 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687144    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 13:15:04.694472 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687146    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 13:15:04.694472 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:04.687149    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 13:15:04.694925 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.687154    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 13:15:04.694925 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.687257    2586 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 13:15:04.694925 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.689172    2586 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 13:15:04.694925 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.690096    2586 server.go:1019] "Starting client certificate rotation"
Sep 18 13:15:04.694925 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.690196    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 13:15:04.694925 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.690844    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 13:15:04.694925 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.692326    2586 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 13:15:04.694925 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.693456    2586 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 13:15:04.697847 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.697832    2586 log.go:25] "Validated CRI v1 runtime API"
Sep 18 13:15:04.699366 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.699351    2586 log.go:25] "Validated CRI v1 image API"
Sep 18 13:15:04.700231 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.700205    2586 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 13:15:04.702419 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.702400    2586 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 cf91f43b-f5b5-44c6-af99-fa1bdf2d8bc7:/dev/nvme0n1p4 db288476-485c-4315-abd6-a5f2d15d49c2:/dev/nvme0n1p3]
Sep 18 13:15:04.702497 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.702418    2586 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 13:15:04.707695 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.707592    2586 manager.go:216] Machine: {Timestamp:2026-09-18 13:15:04.707137164 +0000 UTC m=+2.062596980 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097816 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec251c8ffc7146b7abaf68df5c79fb07 SystemUUID:ec251c8f-fc71-46b7-abaf-68df5c79fb07 BootID:bea14e5c-25e4-4799-afe3-436cbdfaf8f8 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c6:c1:dc:8b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c6:c1:dc:8b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:3e:f5:ea:a0:86 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 13:15:04.707695 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.707688    2586 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 13:15:04.707804 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.707759    2586 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 13:15:04.709264 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.709240    2586 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 13:15:04.709398 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.709265    2586 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-221.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 13:15:04.710392 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.710381    2586 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 13:15:04.710437 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.710395    2586 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 13:15:04.710437 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.710403    2586 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 13:15:04.710437 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.710414    2586 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 13:15:04.710571 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.710484    2586 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 13:15:04.714081 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.714068    2586 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 13:15:04.714190 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.714181    2586 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 13:15:04.714221 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.714214    2586 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 13:15:04.714257 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.714224    2586 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 13:15:04.714257 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.714239    2586 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 13:15:04.714257 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.714249    2586 kubelet.go:398] "Adding apiserver pod source"
Sep 18 13:15:04.714350 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.714264    2586 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 13:15:04.714605 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.714595    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 13:15:04.714646 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.714612    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 13:15:04.718337 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.718311    2586 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 13:15:04.720414 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.720400    2586 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 13:15:04.720465 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.720431    2586 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 13:15:04.721427 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.721407    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 13:15:04.721459 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.721434    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 13:15:04.721459 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.721441    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 13:15:04.721459 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.721449    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 13:15:04.721541 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.721462    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 13:15:04.721541 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.721470    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 13:15:04.721541 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.721485    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 13:15:04.721541 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.721490    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 13:15:04.721541 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.721496    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 13:15:04.721541 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.721502    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 13:15:04.721541 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.721510    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 13:15:04.721541 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.721515    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 13:15:04.721541 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.721542    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 13:15:04.721541 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.721547    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 13:15:04.724957 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.724941    2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 13:15:04.725888 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.725873    2586 server.go:1268] "Started kubelet"
Sep 18 13:15:04.725969 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.725924    2586 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 13:15:04.726013 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.725946    2586 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 13:15:04.726059 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.726017    2586 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 13:15:04.726783 ip-10-0-131-221 systemd[1]: Started Kubernetes Kubelet.
Sep 18 13:15:04.727355 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.726786    2586 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-221.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 13:15:04.727355 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:04.726819    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-221.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 13:15:04.727355 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:04.726887    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 13:15:04.727493 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.727446    2586 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 13:15:04.728575 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.728561    2586 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 13:15:04.730919 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:04.730885    2586 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 13:15:04.732218 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.732207    2586 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 13:15:04.732279 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.732235    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 13:15:04.732329 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:04.732284    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-221.ec2.internal\" not found"
Sep 18 13:15:04.732789 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.732769    2586 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 13:15:04.732789 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.732786    2586 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 13:15:04.732957 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.732818    2586 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 13:15:04.733127 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.733111    2586 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 13:15:04.733127 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.733126    2586 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 13:15:04.733382 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.733367    2586 factory.go:55] Registering systemd factory
Sep 18 13:15:04.733468 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.733386    2586 factory.go:223] Registration of the systemd container factory successfully
Sep 18 13:15:04.733646 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.733629    2586 factory.go:153] Registering CRI-O factory
Sep 18 13:15:04.733706 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.733650    2586 factory.go:223] Registration of the crio container factory successfully
Sep 18 13:15:04.733706 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.733703    2586 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 13:15:04.733789 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.733722    2586 factory.go:103] Registering Raw factory
Sep 18 13:15:04.733789 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.733747    2586 manager.go:1196] Started watching for new ooms in manager
Sep 18 13:15:04.734257 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.734247    2586 manager.go:349] Starting recovery of all containers
Sep 18 13:15:04.735414 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:04.735393    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 13:15:04.735523 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:04.735488    2586 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-221.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 13:15:04.759978 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.759949    2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gjjb2"
Sep 18 13:15:04.761098 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.761070    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gjjb2"
Sep 18 13:15:04.762857 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.762835    2586 manager.go:354] Recovery completed
Sep 18 13:15:04.767564 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.767551    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:15:04.769215 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.769192    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:15:04.769307 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.769227    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:15:04.769307 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.769237    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:15:04.769790 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.769775    2586 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 13:15:04.769790 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.769782    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:15:04.769887 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.769787    2586 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 13:15:04.769887 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.769845    2586 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 13:15:04.770392 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.770379    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:15:04.770435 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.770407    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:15:04.770435 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.770420    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:15:04.771943 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.771927    2586 policy_none.go:49] "None policy: Start"
Sep 18 13:15:04.772003 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.771948    2586 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 13:15:04.772003 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.771961    2586 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 13:15:04.773956 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.773943    2586 policy_none.go:47] "Start"
Sep 18 13:15:04.827980 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.812332    2586 manager.go:337] "Starting Device Plugin manager"
Sep 18 13:15:04.827980 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:04.812365    2586 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 13:15:04.827980 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.812375    2586 server.go:87] "Starting device plugin registration server"
Sep 18 13:15:04.827980 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.812604    2586 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 13:15:04.827980 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.812628    2586 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 13:15:04.827980 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.812694    2586 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 13:15:04.827980 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.812774    2586 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 13:15:04.827980 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.812783    2586 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 13:15:04.827980 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:04.813327    2586 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 13:15:04.827980 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:04.813546    2586 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-221.ec2.internal\" not found"
Sep 18 13:15:04.841752 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.841722    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 13:15:04.843021 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.843005    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 13:15:04.843121 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.843057    2586 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 13:15:04.843121 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.843079    2586 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 13:15:04.843121 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:04.843115    2586 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 13:15:04.845360 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.845343    2586 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:15:04.913358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.913328    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:15:04.914382 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.914366    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:15:04.914490 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.914402    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:15:04.914490 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.914416    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:15:04.914490 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.914445    2586 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-221.ec2.internal"
Sep 18 13:15:04.924145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.924098    2586 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-221.ec2.internal"
Sep 18 13:15:04.924145 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:04.924120    2586 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-221.ec2.internal\": node \"ip-10-0-131-221.ec2.internal\" not found"
Sep 18 13:15:04.942477 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:04.942457    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-221.ec2.internal\" not found"
Sep 18 13:15:04.943611 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.943587    2586 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-221.ec2.internal"]
Sep 18 13:15:04.943671 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.943662    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:15:04.944500 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.944480    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:15:04.944599 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.944517    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:15:04.944599 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.944533    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:15:04.946702 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.946691    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:15:04.946805 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.946791    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal"
Sep 18 13:15:04.946844 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.946820    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:15:04.947386 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.947368    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:15:04.947469 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.947395    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:15:04.947469 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.947409    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:15:04.947469 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.947435    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:15:04.947469 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.947451    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:15:04.947469 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.947462    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:15:04.949496 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.949475    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-221.ec2.internal"
Sep 18 13:15:04.949571 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.949501    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 13:15:04.950182 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.950166    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 13:15:04.950236 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.950194    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 13:15:04.950236 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:04.950204    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeHasSufficientPID"
Sep 18 13:15:04.974743 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:04.974719    2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-221.ec2.internal\" not found" node="ip-10-0-131-221.ec2.internal"
Sep 18 13:15:04.979215 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:04.979197    2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-221.ec2.internal\" not found" node="ip-10-0-131-221.ec2.internal"
Sep 18 13:15:05.035595 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:05.035567    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/31c0bde74e3407e1ddada35fdb2a70e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal\" (UID: \"31c0bde74e3407e1ddada35fdb2a70e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal"
Sep 18 13:15:05.035718 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:05.035602    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31c0bde74e3407e1ddada35fdb2a70e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal\" (UID: \"31c0bde74e3407e1ddada35fdb2a70e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal"
Sep 18 13:15:05.035718 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:05.035621    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/523614729ae9b688582b641ea2fafde6-config\") pod \"kube-apiserver-proxy-ip-10-0-131-221.ec2.internal\" (UID: \"523614729ae9b688582b641ea2fafde6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-221.ec2.internal"
Sep 18 13:15:05.043091 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:05.043069    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-221.ec2.internal\" not found"
Sep 18 13:15:05.136794 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:05.136760    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/31c0bde74e3407e1ddada35fdb2a70e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal\" (UID: \"31c0bde74e3407e1ddada35fdb2a70e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal"
Sep 18 13:15:05.136794 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:05.136795    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31c0bde74e3407e1ddada35fdb2a70e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal\" (UID: \"31c0bde74e3407e1ddada35fdb2a70e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal"
Sep 18 13:15:05.136979 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:05.136812    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/523614729ae9b688582b641ea2fafde6-config\") pod \"kube-apiserver-proxy-ip-10-0-131-221.ec2.internal\" (UID: \"523614729ae9b688582b641ea2fafde6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-221.ec2.internal"
Sep 18 13:15:05.136979 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:05.136846    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/31c0bde74e3407e1ddada35fdb2a70e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal\" (UID: \"31c0bde74e3407e1ddada35fdb2a70e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal"
Sep 18 13:15:05.136979 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:05.136853    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/31c0bde74e3407e1ddada35fdb2a70e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal\" (UID: \"31c0bde74e3407e1ddada35fdb2a70e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal"
Sep 18 13:15:05.136979 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:05.136888    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/523614729ae9b688582b641ea2fafde6-config\") pod \"kube-apiserver-proxy-ip-10-0-131-221.ec2.internal\" (UID: \"523614729ae9b688582b641ea2fafde6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-221.ec2.internal"
Sep 18 13:15:05.143863 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:05.143844    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-221.ec2.internal\" not found"
Sep 18 13:15:05.244726 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:05.244648    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-221.ec2.internal\" not found"
Sep 18 13:15:05.277819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:05.277787    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal"
Sep 18 13:15:05.283531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:05.282754    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-221.ec2.internal"
Sep 18 13:15:05.345226 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:05.345196    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-221.ec2.internal\" not found"
Sep 18 13:15:05.445712 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:05.445684    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-221.ec2.internal\" not found"
Sep 18 13:15:05.546239 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:05.546171    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-221.ec2.internal\" not found"
Sep 18 13:15:05.646802 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:05.646772    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-221.ec2.internal\" not found"
Sep 18 13:15:05.689967 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:05.689937    2586 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 13:15:05.690359 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:05.690092    2586 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 13:15:05.724927 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:05.724894    2586 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:15:05.732326 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:05.732301    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 13:15:05.732772 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:05.732755    2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal"
Sep 18 13:15:05.752293 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:05.752267    2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 13:15:05.753823 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:05.753806    2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-221.ec2.internal"
Sep 18 13:15:05.756732 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:05.756716    2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 13:15:05.766435 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:05.766412    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 13:10:04 +0000 UTC" deadline="2028-04-21 11:29:56.040902401 +0000 UTC"
Sep 18 13:15:05.766435 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:05.766433    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13942h14m50.274472654s"
Sep 18 13:15:05.771493 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:05.771479    2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 13:15:05.835812 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:05.835752    2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-t7cbv"
Sep 18 13:15:05.847356 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:05.847334    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-t7cbv"
Sep 18 13:15:05.972252 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:05.972216    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c0bde74e3407e1ddada35fdb2a70e7.slice/crio-f4046cb28f63fdf9af32ae8b0709c608c7746c168f4820c0789fb26ce07c6462 WatchSource:0}: Error finding container f4046cb28f63fdf9af32ae8b0709c608c7746c168f4820c0789fb26ce07c6462: Status 404 returned error can't find the container with id f4046cb28f63fdf9af32ae8b0709c608c7746c168f4820c0789fb26ce07c6462
Sep 18 13:15:05.972557 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:05.972537    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523614729ae9b688582b641ea2fafde6.slice/crio-c0890bb7afa93c863dbed9527680932a1bf5a5fcc9158d1e4398a034cdcee42f WatchSource:0}: Error finding container c0890bb7afa93c863dbed9527680932a1bf5a5fcc9158d1e4398a034cdcee42f: Status 404 returned error can't find the container with id c0890bb7afa93c863dbed9527680932a1bf5a5fcc9158d1e4398a034cdcee42f
Sep 18 13:15:05.976970 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:05.976954    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 13:15:06.053167 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.053138    2586 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:15:06.176700 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.176672    2586 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:15:06.715527 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.715495    2586 apiserver.go:52] "Watching apiserver"
Sep 18 13:15:06.722769 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.722722    2586 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 13:15:06.723521 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.723495    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-tmdn8","openshift-network-operator/iptables-alerter-5td8r","openshift-ovn-kubernetes/ovnkube-node-lcqcw","kube-system/konnectivity-agent-fmxrm","openshift-cluster-node-tuning-operator/tuned-jxbpd","openshift-dns/node-resolver-lbfgj","openshift-image-registry/node-ca-s99br","openshift-multus/multus-additional-cni-plugins-kv86r","openshift-multus/multus-vsgwg","kube-system/global-pull-secret-syncer-pc5zc","kube-system/kube-apiserver-proxy-ip-10-0-131-221.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal","openshift-multus/network-metrics-daemon-qlzxp"]
Sep 18 13:15:06.726274 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.726256    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s99br"
Sep 18 13:15:06.728596 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.728238    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5td8r"
Sep 18 13:15:06.731258 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.731236    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.733399 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.733380    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fmxrm"
Sep 18 13:15:06.735629 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.735609    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:06.735721 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:06.735678    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlzxp" podUID="47797f8b-a736-4caa-9b4a-6de71b38c001"
Sep 18 13:15:06.738167 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.738151    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc"
Sep 18 13:15:06.740412 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.740393    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.742663 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.742645    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:06.742768 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:06.742716    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmdn8" podUID="1b7871d5-8610-4b4a-b099-4163ff75df5e"
Sep 18 13:15:06.745068 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.744979    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:06.745068 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:06.745036    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pc5zc" podUID="7e310da4-ab19-4631-9717-86fb07925256"
Sep 18 13:15:06.746109 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746089    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-var-lib-openvswitch\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.746207 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746123    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-etc-openvswitch\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.746207 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746148    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-run-openvswitch\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.746207 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746172    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-run-ovn\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.746364 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746219    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-host-cni-netd\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.746364 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746242    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1cc32441-1189-4113-bf17-2769fa23a747-env-overrides\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.746364 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746264    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-host-kubelet\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.746364 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746287    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e53a6d50-eb67-499c-9626-d7788ef58cef-konnectivity-ca\") pod \"konnectivity-agent-fmxrm\" (UID: \"e53a6d50-eb67-499c-9626-d7788ef58cef\") " pod="kube-system/konnectivity-agent-fmxrm"
Sep 18 13:15:06.746364 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746339    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b6e6324a-7c14-4edb-9e82-0729e23ce5b8-host-slash\") pod \"iptables-alerter-5td8r\" (UID: \"b6e6324a-7c14-4edb-9e82-0729e23ce5b8\") " pod="openshift-network-operator/iptables-alerter-5td8r"
Sep 18 13:15:06.746602 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746381    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22j7f\" (UniqueName: \"kubernetes.io/projected/b6e6324a-7c14-4edb-9e82-0729e23ce5b8-kube-api-access-22j7f\") pod \"iptables-alerter-5td8r\" (UID: \"b6e6324a-7c14-4edb-9e82-0729e23ce5b8\") " pod="openshift-network-operator/iptables-alerter-5td8r"
Sep 18 13:15:06.746602 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746405    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-systemd-units\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.746602 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746427    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvsf5\" (UniqueName: \"kubernetes.io/projected/7721f73b-5184-4404-b65b-774dfff4df76-kube-api-access-bvsf5\") pod \"node-ca-s99br\" (UID: \"7721f73b-5184-4404-b65b-774dfff4df76\") " pod="openshift-image-registry/node-ca-s99br"
Sep 18 13:15:06.746602 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746451    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-log-socket\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.746602 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746473    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-host-run-ovn-kubernetes\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.746602 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746492    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1cc32441-1189-4113-bf17-2769fa23a747-ovn-node-metrics-cert\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.746602 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746505    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7721f73b-5184-4404-b65b-774dfff4df76-serviceca\") pod \"node-ca-s99br\" (UID: \"7721f73b-5184-4404-b65b-774dfff4df76\") " pod="openshift-image-registry/node-ca-s99br"
Sep 18 13:15:06.746602 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746519    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-run-systemd\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.746602 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746533    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e53a6d50-eb67-499c-9626-d7788ef58cef-agent-certs\") pod \"konnectivity-agent-fmxrm\" (UID: \"e53a6d50-eb67-499c-9626-d7788ef58cef\") " pod="kube-system/konnectivity-agent-fmxrm"
Sep 18 13:15:06.746602 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746560    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b6e6324a-7c14-4edb-9e82-0729e23ce5b8-iptables-alerter-script\") pod \"iptables-alerter-5td8r\" (UID: \"b6e6324a-7c14-4edb-9e82-0729e23ce5b8\") " pod="openshift-network-operator/iptables-alerter-5td8r"
Sep 18 13:15:06.746602 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746573    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-host-cni-bin\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.746602 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746589    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1cc32441-1189-4113-bf17-2769fa23a747-ovnkube-config\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.747155 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746612    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.747155 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746627    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1cc32441-1189-4113-bf17-2769fa23a747-ovnkube-script-lib\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.747155 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746640    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7721f73b-5184-4404-b65b-774dfff4df76-host\") pod \"node-ca-s99br\" (UID: \"7721f73b-5184-4404-b65b-774dfff4df76\") " pod="openshift-image-registry/node-ca-s99br"
Sep 18 13:15:06.747155 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746653    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-host-slash\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.747155 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746671    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-host-run-netns\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.747155 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746686    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-node-log\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.747155 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.746699    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dwr2\" (UniqueName: \"kubernetes.io/projected/1cc32441-1189-4113-bf17-2769fa23a747-kube-api-access-4dwr2\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.748148 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.747599    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lbfgj"
Sep 18 13:15:06.749985 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.749965    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kv86r"
Sep 18 13:15:06.752275 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.752252    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.754699 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.754675    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 13:15:06.755270 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.755250    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:15:06.756053 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.756033    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 13:15:06.756150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.756100    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 13:15:06.756313 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.756287    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 13:15:06.756539 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.756392    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 13:15:06.756539 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.756501    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 13:15:06.757078 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.757052    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 13:15:06.757162 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.757134    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 13:15:06.757397 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.757305    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 13:15:06.757452 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.757394    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wfpgq\""
Sep 18 13:15:06.757489 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.757471    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 13:15:06.757676 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.757659    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 13:15:06.757876 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.757858    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 13:15:06.758096 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.758078    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 13:15:06.758096 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.758088    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 13:15:06.758216 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.758103    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 13:15:06.758269 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.758228    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 13:15:06.758405 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.758381    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 13:15:06.758466 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.758429    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 13:15:06.758582 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.758566    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zb4hw\""
Sep 18 13:15:06.761711 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.761521    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 13:15:06.761711 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.761570    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zxldr\""
Sep 18 13:15:06.761711 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.761570    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 13:15:06.761711 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.761711    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 13:15:06.763487 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.762527    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 13:15:06.763487 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.762636    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6zks5\""
Sep 18 13:15:06.763487 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.762746    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qcxdz\""
Sep 18 13:15:06.763487 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.762851    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 13:15:06.763487 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.762923    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vkb2b\""
Sep 18 13:15:06.763487 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.763148    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:15:06.763487 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.763389    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-krbtm\""
Sep 18 13:15:06.768295 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.767410    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 13:15:06.768295 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.767691    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2blz6\""
Sep 18 13:15:06.768295 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.768064    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9rlwz\""
Sep 18 13:15:06.769782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.769763    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 13:15:06.834675 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.834639    2586 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 13:15:06.849358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.846833    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-host\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.849358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.846875    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-run-openvswitch\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.849358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.846923    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-run-ovn\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.849358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.846948    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-host-cni-netd\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.849358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.846970    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1cc32441-1189-4113-bf17-2769fa23a747-env-overrides\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.849358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.846994    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-cnibin\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.849358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847020    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e53a6d50-eb67-499c-9626-d7788ef58cef-konnectivity-ca\") pod \"konnectivity-agent-fmxrm\" (UID: \"e53a6d50-eb67-499c-9626-d7788ef58cef\") " pod="kube-system/konnectivity-agent-fmxrm"
Sep 18 13:15:06.849358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847038    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b72095e4-8ae2-46ee-83a2-77a996aa45bd-etc-tuned\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.849358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847052    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b72095e4-8ae2-46ee-83a2-77a996aa45bd-tmp\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.849358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847068    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj28r\" (UniqueName: \"kubernetes.io/projected/af23cea6-b0da-4b1c-a704-d24956242e46-kube-api-access-nj28r\") pod \"multus-additional-cni-plugins-kv86r\" (UID: \"af23cea6-b0da-4b1c-a704-d24956242e46\") " pod="openshift-multus/multus-additional-cni-plugins-kv86r"
Sep 18 13:15:06.849358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847083    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22j7f\" (UniqueName: \"kubernetes.io/projected/b6e6324a-7c14-4edb-9e82-0729e23ce5b8-kube-api-access-22j7f\") pod \"iptables-alerter-5td8r\" (UID: \"b6e6324a-7c14-4edb-9e82-0729e23ce5b8\") " pod="openshift-network-operator/iptables-alerter-5td8r"
Sep 18 13:15:06.849358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847098    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-systemd-units\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.849358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847113    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-host-run-multus-certs\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.849358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847127    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-etc-kubernetes\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.849358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847145    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9qpn\" (UniqueName: \"kubernetes.io/projected/73749cf1-5bc4-452a-b1a4-b3ef009e3ff3-kube-api-access-h9qpn\") pod \"node-resolver-lbfgj\" (UID: \"73749cf1-5bc4-452a-b1a4-b3ef009e3ff3\") " pod="openshift-dns/node-resolver-lbfgj"
Sep 18 13:15:06.849358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847158    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-etc-kubernetes\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.849358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847172    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-etc-sysctl-conf\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.850208 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847191    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-etc-systemd\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.850208 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847212    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-host-run-netns\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.850208 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847247    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-sys\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.850208 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847270    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47797f8b-a736-4caa-9b4a-6de71b38c001-metrics-certs\") pod \"network-metrics-daemon-qlzxp\" (UID: \"47797f8b-a736-4caa-9b4a-6de71b38c001\") " pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:06.850208 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847294    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgmvf\" (UniqueName: \"kubernetes.io/projected/990bd81d-70fa-437e-8c1c-2521d4403925-kube-api-access-lgmvf\") pod \"aws-ebs-csi-driver-node-4f7mc\" (UID: \"990bd81d-70fa-437e-8c1c-2521d4403925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc"
Sep 18 13:15:06.850208 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847323    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af23cea6-b0da-4b1c-a704-d24956242e46-system-cni-dir\") pod \"multus-additional-cni-plugins-kv86r\" (UID: \"af23cea6-b0da-4b1c-a704-d24956242e46\") " pod="openshift-multus/multus-additional-cni-plugins-kv86r"
Sep 18 13:15:06.850208 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847350    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/af23cea6-b0da-4b1c-a704-d24956242e46-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kv86r\" (UID: \"af23cea6-b0da-4b1c-a704-d24956242e46\") " pod="openshift-multus/multus-additional-cni-plugins-kv86r"
Sep 18 13:15:06.850208 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847376    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7721f73b-5184-4404-b65b-774dfff4df76-serviceca\") pod \"node-ca-s99br\" (UID: \"7721f73b-5184-4404-b65b-774dfff4df76\") " pod="openshift-image-registry/node-ca-s99br"
Sep 18 13:15:06.850208 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847399    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-run-systemd\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.850208 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847431    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e53a6d50-eb67-499c-9626-d7788ef58cef-agent-certs\") pod \"konnectivity-agent-fmxrm\" (UID: \"e53a6d50-eb67-499c-9626-d7788ef58cef\") " pod="kube-system/konnectivity-agent-fmxrm"
Sep 18 13:15:06.850208 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847454    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-etc-sysconfig\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.850208 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847472    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-host-cni-bin\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.850208 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847487    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1cc32441-1189-4113-bf17-2769fa23a747-ovnkube-config\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.850208 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847501    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-multus-socket-dir-parent\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.850208 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847520    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/990bd81d-70fa-437e-8c1c-2521d4403925-device-dir\") pod \"aws-ebs-csi-driver-node-4f7mc\" (UID: \"990bd81d-70fa-437e-8c1c-2521d4403925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc"
Sep 18 13:15:06.850208 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847539    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e310da4-ab19-4631-9717-86fb07925256-kubelet-config\") pod \"global-pull-secret-syncer-pc5zc\" (UID: \"7e310da4-ab19-4631-9717-86fb07925256\") " pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:06.850208 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847552    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e310da4-ab19-4631-9717-86fb07925256-dbus\") pod \"global-pull-secret-syncer-pc5zc\" (UID: \"7e310da4-ab19-4631-9717-86fb07925256\") " pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:06.850782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847550    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-run-openvswitch\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.850782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847576    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/73749cf1-5bc4-452a-b1a4-b3ef009e3ff3-hosts-file\") pod \"node-resolver-lbfgj\" (UID: \"73749cf1-5bc4-452a-b1a4-b3ef009e3ff3\") " pod="openshift-dns/node-resolver-lbfgj"
Sep 18 13:15:06.850782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847597    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-run-ovn\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.850782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847605    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-etc-modprobe-d\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.850782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847620    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-595h9\" (UniqueName: \"kubernetes.io/projected/b72095e4-8ae2-46ee-83a2-77a996aa45bd-kube-api-access-595h9\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.850782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847638    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-host-cni-netd\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.850782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847643    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/566608dd-e846-4eec-9f7a-9596f671090e-multus-daemon-config\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.850782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847658    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73749cf1-5bc4-452a-b1a4-b3ef009e3ff3-tmp-dir\") pod \"node-resolver-lbfgj\" (UID: \"73749cf1-5bc4-452a-b1a4-b3ef009e3ff3\") " pod="openshift-dns/node-resolver-lbfgj"
Sep 18 13:15:06.850782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847674    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-etc-sysctl-d\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.850782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847689    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-host-run-netns\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.850782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847704    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-node-log\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.850782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847719    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dwr2\" (UniqueName: \"kubernetes.io/projected/1cc32441-1189-4113-bf17-2769fa23a747-kube-api-access-4dwr2\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.850782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847733    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-host-run-k8s-cni-cncf-io\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.850782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847778    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-host-var-lib-cni-bin\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.850782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847809    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-var-lib-openvswitch\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.850782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847834    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-etc-openvswitch\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.850782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847871    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-lib-modules\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.851795 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847917    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx7pn\" (UniqueName: \"kubernetes.io/projected/47797f8b-a736-4caa-9b4a-6de71b38c001-kube-api-access-jx7pn\") pod \"network-metrics-daemon-qlzxp\" (UID: \"47797f8b-a736-4caa-9b4a-6de71b38c001\") " pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:06.851795 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847945    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/990bd81d-70fa-437e-8c1c-2521d4403925-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4f7mc\" (UID: \"990bd81d-70fa-437e-8c1c-2521d4403925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc"
Sep 18 13:15:06.851795 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847967    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/990bd81d-70fa-437e-8c1c-2521d4403925-sys-fs\") pod \"aws-ebs-csi-driver-node-4f7mc\" (UID: \"990bd81d-70fa-437e-8c1c-2521d4403925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc"
Sep 18 13:15:06.851795 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.847993    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/af23cea6-b0da-4b1c-a704-d24956242e46-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kv86r\" (UID: \"af23cea6-b0da-4b1c-a704-d24956242e46\") " pod="openshift-multus/multus-additional-cni-plugins-kv86r"
Sep 18 13:15:06.851795 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.848018    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-host-kubelet\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.851795 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.848033    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/566608dd-e846-4eec-9f7a-9596f671090e-cni-binary-copy\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.851795 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.848037    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1cc32441-1189-4113-bf17-2769fa23a747-env-overrides\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.851795 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.848052    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-run\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.851795 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.848081    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b6e6324a-7c14-4edb-9e82-0729e23ce5b8-host-slash\") pod \"iptables-alerter-5td8r\" (UID: \"b6e6324a-7c14-4edb-9e82-0729e23ce5b8\") " pod="openshift-network-operator/iptables-alerter-5td8r"
Sep 18 13:15:06.851795 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.848098    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-systemd-units\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.851795 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.848101    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt4jz\" (UniqueName: \"kubernetes.io/projected/566608dd-e846-4eec-9f7a-9596f671090e-kube-api-access-zt4jz\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.851795 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.848294    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b6e6324a-7c14-4edb-9e82-0729e23ce5b8-host-slash\") pod \"iptables-alerter-5td8r\" (UID: \"b6e6324a-7c14-4edb-9e82-0729e23ce5b8\") " pod="openshift-network-operator/iptables-alerter-5td8r"
Sep 18 13:15:06.851795 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.848329    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-host-run-netns\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.851795 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.848362    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-node-log\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.851795 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.848562    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7721f73b-5184-4404-b65b-774dfff4df76-serviceca\") pod \"node-ca-s99br\" (UID: \"7721f73b-5184-4404-b65b-774dfff4df76\") " pod="openshift-image-registry/node-ca-s99br"
Sep 18 13:15:06.851795 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.848566    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-var-lib-openvswitch\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.851795 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.848606    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-host-cni-bin\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.852614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.848623    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-run-systemd\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.852614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.848663    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-etc-openvswitch\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.852614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.848942    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1cc32441-1189-4113-bf17-2769fa23a747-ovnkube-config\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.852614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.848975    2586 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 13:15:06.852614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849018    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvsf5\" (UniqueName: \"kubernetes.io/projected/7721f73b-5184-4404-b65b-774dfff4df76-kube-api-access-bvsf5\") pod \"node-ca-s99br\" (UID: \"7721f73b-5184-4404-b65b-774dfff4df76\") " pod="openshift-image-registry/node-ca-s99br"
Sep 18 13:15:06.852614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849032    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-host-kubelet\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.852614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849057    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-log-socket\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.852614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849158    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-host-run-ovn-kubernetes\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.852614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849180    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-multus-cni-dir\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.852614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849230    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-host-run-ovn-kubernetes\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.852614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849234    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-log-socket\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.852614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849242    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/990bd81d-70fa-437e-8c1c-2521d4403925-registration-dir\") pod \"aws-ebs-csi-driver-node-4f7mc\" (UID: \"990bd81d-70fa-437e-8c1c-2521d4403925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc"
Sep 18 13:15:06.852614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849295    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/990bd81d-70fa-437e-8c1c-2521d4403925-etc-selinux\") pod \"aws-ebs-csi-driver-node-4f7mc\" (UID: \"990bd81d-70fa-437e-8c1c-2521d4403925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc"
Sep 18 13:15:06.852614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849336    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88wcc\" (UniqueName: \"kubernetes.io/projected/1b7871d5-8610-4b4a-b099-4163ff75df5e-kube-api-access-88wcc\") pod \"network-check-target-tmdn8\" (UID: \"1b7871d5-8610-4b4a-b099-4163ff75df5e\") " pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:06.852614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849362    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af23cea6-b0da-4b1c-a704-d24956242e46-cnibin\") pod \"multus-additional-cni-plugins-kv86r\" (UID: \"af23cea6-b0da-4b1c-a704-d24956242e46\") " pod="openshift-multus/multus-additional-cni-plugins-kv86r"
Sep 18 13:15:06.852614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849385    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal" event={"ID":"31c0bde74e3407e1ddada35fdb2a70e7","Type":"ContainerStarted","Data":"f4046cb28f63fdf9af32ae8b0709c608c7746c168f4820c0789fb26ce07c6462"}
Sep 18 13:15:06.852614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849441    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1cc32441-1189-4113-bf17-2769fa23a747-ovn-node-metrics-cert\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.853371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849486    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e53a6d50-eb67-499c-9626-d7788ef58cef-konnectivity-ca\") pod \"konnectivity-agent-fmxrm\" (UID: \"e53a6d50-eb67-499c-9626-d7788ef58cef\") " pod="kube-system/konnectivity-agent-fmxrm"
Sep 18 13:15:06.853371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849489    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-host-var-lib-kubelet\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.853371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849564    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-multus-conf-dir\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.853371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849586    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/990bd81d-70fa-437e-8c1c-2521d4403925-socket-dir\") pod \"aws-ebs-csi-driver-node-4f7mc\" (UID: \"990bd81d-70fa-437e-8c1c-2521d4403925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc"
Sep 18 13:15:06.853371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849608    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af23cea6-b0da-4b1c-a704-d24956242e46-cni-binary-copy\") pod \"multus-additional-cni-plugins-kv86r\" (UID: \"af23cea6-b0da-4b1c-a704-d24956242e46\") " pod="openshift-multus/multus-additional-cni-plugins-kv86r"
Sep 18 13:15:06.853371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849636    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b6e6324a-7c14-4edb-9e82-0729e23ce5b8-iptables-alerter-script\") pod \"iptables-alerter-5td8r\" (UID: \"b6e6324a-7c14-4edb-9e82-0729e23ce5b8\") " pod="openshift-network-operator/iptables-alerter-5td8r"
Sep 18 13:15:06.853371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849657    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-hostroot\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.853371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849693    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 13:10:05 +0000 UTC" deadline="2028-02-28 12:37:29.437668552 +0000 UTC"
Sep 18 13:15:06.853371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849727    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12671h22m22.587944582s"
Sep 18 13:15:06.853371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849721    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e310da4-ab19-4631-9717-86fb07925256-original-pull-secret\") pod \"global-pull-secret-syncer-pc5zc\" (UID: \"7e310da4-ab19-4631-9717-86fb07925256\") " pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:06.853371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849783    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.853371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849820    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1cc32441-1189-4113-bf17-2769fa23a747-ovnkube-script-lib\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.853371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849851    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.853371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849882    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-system-cni-dir\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.853371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.849953    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af23cea6-b0da-4b1c-a704-d24956242e46-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kv86r\" (UID: \"af23cea6-b0da-4b1c-a704-d24956242e46\") " pod="openshift-multus/multus-additional-cni-plugins-kv86r"
Sep 18 13:15:06.853371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.850001    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7721f73b-5184-4404-b65b-774dfff4df76-host\") pod \"node-ca-s99br\" (UID: \"7721f73b-5184-4404-b65b-774dfff4df76\") " pod="openshift-image-registry/node-ca-s99br"
Sep 18 13:15:06.853371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.850037    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-host-slash\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.853371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.850069    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7721f73b-5184-4404-b65b-774dfff4df76-host\") pod \"node-ca-s99br\" (UID: \"7721f73b-5184-4404-b65b-774dfff4df76\") " pod="openshift-image-registry/node-ca-s99br"
Sep 18 13:15:06.854316 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.850116    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-host-var-lib-cni-multus\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.854316 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.850128    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b6e6324a-7c14-4edb-9e82-0729e23ce5b8-iptables-alerter-script\") pod \"iptables-alerter-5td8r\" (UID: \"b6e6324a-7c14-4edb-9e82-0729e23ce5b8\") " pod="openshift-network-operator/iptables-alerter-5td8r"
Sep 18 13:15:06.854316 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.850133    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1cc32441-1189-4113-bf17-2769fa23a747-host-slash\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.854316 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.850172    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-var-lib-kubelet\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.854316 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.850364    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1cc32441-1189-4113-bf17-2769fa23a747-ovnkube-script-lib\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.854316 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.851775    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-221.ec2.internal" event={"ID":"523614729ae9b688582b641ea2fafde6","Type":"ContainerStarted","Data":"c0890bb7afa93c863dbed9527680932a1bf5a5fcc9158d1e4398a034cdcee42f"}
Sep 18 13:15:06.854316 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.853611    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1cc32441-1189-4113-bf17-2769fa23a747-ovn-node-metrics-cert\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.854316 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.853709    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e53a6d50-eb67-499c-9626-d7788ef58cef-agent-certs\") pod \"konnectivity-agent-fmxrm\" (UID: \"e53a6d50-eb67-499c-9626-d7788ef58cef\") " pod="kube-system/konnectivity-agent-fmxrm"
Sep 18 13:15:06.863645 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.863619    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvsf5\" (UniqueName: \"kubernetes.io/projected/7721f73b-5184-4404-b65b-774dfff4df76-kube-api-access-bvsf5\") pod \"node-ca-s99br\" (UID: \"7721f73b-5184-4404-b65b-774dfff4df76\") " pod="openshift-image-registry/node-ca-s99br"
Sep 18 13:15:06.864419 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.864396    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22j7f\" (UniqueName: \"kubernetes.io/projected/b6e6324a-7c14-4edb-9e82-0729e23ce5b8-kube-api-access-22j7f\") pod \"iptables-alerter-5td8r\" (UID: \"b6e6324a-7c14-4edb-9e82-0729e23ce5b8\") " pod="openshift-network-operator/iptables-alerter-5td8r"
Sep 18 13:15:06.864600 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.864581    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dwr2\" (UniqueName: \"kubernetes.io/projected/1cc32441-1189-4113-bf17-2769fa23a747-kube-api-access-4dwr2\") pod \"ovnkube-node-lcqcw\" (UID: \"1cc32441-1189-4113-bf17-2769fa23a747\") " pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:06.950409 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950372    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e310da4-ab19-4631-9717-86fb07925256-kubelet-config\") pod \"global-pull-secret-syncer-pc5zc\" (UID: \"7e310da4-ab19-4631-9717-86fb07925256\") " pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:06.950409 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950410    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e310da4-ab19-4631-9717-86fb07925256-dbus\") pod \"global-pull-secret-syncer-pc5zc\" (UID: \"7e310da4-ab19-4631-9717-86fb07925256\") " pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:06.950629 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950433    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/73749cf1-5bc4-452a-b1a4-b3ef009e3ff3-hosts-file\") pod \"node-resolver-lbfgj\" (UID: \"73749cf1-5bc4-452a-b1a4-b3ef009e3ff3\") " pod="openshift-dns/node-resolver-lbfgj"
Sep 18 13:15:06.950629 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950454    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-etc-modprobe-d\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.950629 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950473    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-595h9\" (UniqueName: \"kubernetes.io/projected/b72095e4-8ae2-46ee-83a2-77a996aa45bd-kube-api-access-595h9\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.950629 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950494    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/566608dd-e846-4eec-9f7a-9596f671090e-multus-daemon-config\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.950629 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950516    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73749cf1-5bc4-452a-b1a4-b3ef009e3ff3-tmp-dir\") pod \"node-resolver-lbfgj\" (UID: \"73749cf1-5bc4-452a-b1a4-b3ef009e3ff3\") " pod="openshift-dns/node-resolver-lbfgj"
Sep 18 13:15:06.950629 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950536    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-etc-sysctl-d\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.950629 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950534    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/73749cf1-5bc4-452a-b1a4-b3ef009e3ff3-hosts-file\") pod \"node-resolver-lbfgj\" (UID: \"73749cf1-5bc4-452a-b1a4-b3ef009e3ff3\") " pod="openshift-dns/node-resolver-lbfgj"
Sep 18 13:15:06.950629 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950558    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-host-run-k8s-cni-cncf-io\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.950629 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950560    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7e310da4-ab19-4631-9717-86fb07925256-kubelet-config\") pod \"global-pull-secret-syncer-pc5zc\" (UID: \"7e310da4-ab19-4631-9717-86fb07925256\") " pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:06.950629 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950601    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-host-run-k8s-cni-cncf-io\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.951122 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950638    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-etc-modprobe-d\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.951122 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950637    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7e310da4-ab19-4631-9717-86fb07925256-dbus\") pod \"global-pull-secret-syncer-pc5zc\" (UID: \"7e310da4-ab19-4631-9717-86fb07925256\") " pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:06.951122 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950668    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-host-var-lib-cni-bin\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.951122 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950697    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-lib-modules\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.951122 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950719    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx7pn\" (UniqueName: \"kubernetes.io/projected/47797f8b-a736-4caa-9b4a-6de71b38c001-kube-api-access-jx7pn\") pod \"network-metrics-daemon-qlzxp\" (UID: \"47797f8b-a736-4caa-9b4a-6de71b38c001\") " pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:06.951122 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950744    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/990bd81d-70fa-437e-8c1c-2521d4403925-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4f7mc\" (UID: \"990bd81d-70fa-437e-8c1c-2521d4403925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc"
Sep 18 13:15:06.951122 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950769    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/990bd81d-70fa-437e-8c1c-2521d4403925-sys-fs\") pod \"aws-ebs-csi-driver-node-4f7mc\" (UID: \"990bd81d-70fa-437e-8c1c-2521d4403925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc"
Sep 18 13:15:06.951122 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950797    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/af23cea6-b0da-4b1c-a704-d24956242e46-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kv86r\" (UID: \"af23cea6-b0da-4b1c-a704-d24956242e46\") " pod="openshift-multus/multus-additional-cni-plugins-kv86r"
Sep 18 13:15:06.951122 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950821    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/566608dd-e846-4eec-9f7a-9596f671090e-cni-binary-copy\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.951122 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950844    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-run\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.951122 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950846    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73749cf1-5bc4-452a-b1a4-b3ef009e3ff3-tmp-dir\") pod \"node-resolver-lbfgj\" (UID: \"73749cf1-5bc4-452a-b1a4-b3ef009e3ff3\") " pod="openshift-dns/node-resolver-lbfgj"
Sep 18 13:15:06.951122 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950867    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt4jz\" (UniqueName: \"kubernetes.io/projected/566608dd-e846-4eec-9f7a-9596f671090e-kube-api-access-zt4jz\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.951122 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950895    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-multus-cni-dir\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.951122 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950940    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/990bd81d-70fa-437e-8c1c-2521d4403925-registration-dir\") pod \"aws-ebs-csi-driver-node-4f7mc\" (UID: \"990bd81d-70fa-437e-8c1c-2521d4403925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc"
Sep 18 13:15:06.951122 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950944    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-host-var-lib-cni-bin\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.951122 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950962    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/990bd81d-70fa-437e-8c1c-2521d4403925-etc-selinux\") pod \"aws-ebs-csi-driver-node-4f7mc\" (UID: \"990bd81d-70fa-437e-8c1c-2521d4403925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc"
Sep 18 13:15:06.951122 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950987    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88wcc\" (UniqueName: \"kubernetes.io/projected/1b7871d5-8610-4b4a-b099-4163ff75df5e-kube-api-access-88wcc\") pod \"network-check-target-tmdn8\" (UID: \"1b7871d5-8610-4b4a-b099-4163ff75df5e\") " pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:06.951932 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951019    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af23cea6-b0da-4b1c-a704-d24956242e46-cnibin\") pod \"multus-additional-cni-plugins-kv86r\" (UID: \"af23cea6-b0da-4b1c-a704-d24956242e46\") " pod="openshift-multus/multus-additional-cni-plugins-kv86r"
Sep 18 13:15:06.951932 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951034    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/990bd81d-70fa-437e-8c1c-2521d4403925-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4f7mc\" (UID: \"990bd81d-70fa-437e-8c1c-2521d4403925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc"
Sep 18 13:15:06.951932 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951044    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-host-var-lib-kubelet\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.951932 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951063    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-lib-modules\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.951932 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951080    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-multus-conf-dir\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.951932 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951088    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-host-var-lib-kubelet\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.951932 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951107    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/990bd81d-70fa-437e-8c1c-2521d4403925-socket-dir\") pod \"aws-ebs-csi-driver-node-4f7mc\" (UID: \"990bd81d-70fa-437e-8c1c-2521d4403925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc"
Sep 18 13:15:06.951932 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951131    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af23cea6-b0da-4b1c-a704-d24956242e46-cni-binary-copy\") pod \"multus-additional-cni-plugins-kv86r\" (UID: \"af23cea6-b0da-4b1c-a704-d24956242e46\") " pod="openshift-multus/multus-additional-cni-plugins-kv86r"
Sep 18 13:15:06.951932 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951152    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/566608dd-e846-4eec-9f7a-9596f671090e-multus-daemon-config\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.951932 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951140    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/990bd81d-70fa-437e-8c1c-2521d4403925-sys-fs\") pod \"aws-ebs-csi-driver-node-4f7mc\" (UID: \"990bd81d-70fa-437e-8c1c-2521d4403925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc"
Sep 18 13:15:06.951932 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951179    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-hostroot\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.951932 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951228    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-multus-conf-dir\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.951932 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951231    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/990bd81d-70fa-437e-8c1c-2521d4403925-registration-dir\") pod \"aws-ebs-csi-driver-node-4f7mc\" (UID: \"990bd81d-70fa-437e-8c1c-2521d4403925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc"
Sep 18 13:15:06.951932 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951259    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e310da4-ab19-4631-9717-86fb07925256-original-pull-secret\") pod \"global-pull-secret-syncer-pc5zc\" (UID: \"7e310da4-ab19-4631-9717-86fb07925256\") " pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:06.951932 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951289    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-system-cni-dir\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.951932 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951330    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af23cea6-b0da-4b1c-a704-d24956242e46-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kv86r\" (UID: \"af23cea6-b0da-4b1c-a704-d24956242e46\") " pod="openshift-multus/multus-additional-cni-plugins-kv86r"
Sep 18 13:15:06.951932 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951333    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/990bd81d-70fa-437e-8c1c-2521d4403925-socket-dir\") pod \"aws-ebs-csi-driver-node-4f7mc\" (UID: \"990bd81d-70fa-437e-8c1c-2521d4403925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc"
Sep 18 13:15:06.952672 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951345    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-run\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.952672 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951370    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-hostroot\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.952672 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951400    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-host-var-lib-cni-multus\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.952672 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951425    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-var-lib-kubelet\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.952672 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951447    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-host\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.952672 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951475    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-cnibin\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.952672 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951499    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b72095e4-8ae2-46ee-83a2-77a996aa45bd-etc-tuned\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.952672 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951521    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b72095e4-8ae2-46ee-83a2-77a996aa45bd-tmp\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.952672 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951545    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj28r\" (UniqueName: \"kubernetes.io/projected/af23cea6-b0da-4b1c-a704-d24956242e46-kube-api-access-nj28r\") pod \"multus-additional-cni-plugins-kv86r\" (UID: \"af23cea6-b0da-4b1c-a704-d24956242e46\") " pod="openshift-multus/multus-additional-cni-plugins-kv86r"
Sep 18 13:15:06.952672 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951572    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-host-run-multus-certs\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.952672 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951618    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-etc-kubernetes\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.952672 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951643    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9qpn\" (UniqueName: \"kubernetes.io/projected/73749cf1-5bc4-452a-b1a4-b3ef009e3ff3-kube-api-access-h9qpn\") pod \"node-resolver-lbfgj\" (UID: \"73749cf1-5bc4-452a-b1a4-b3ef009e3ff3\") " pod="openshift-dns/node-resolver-lbfgj"
Sep 18 13:15:06.952672 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951668    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-etc-kubernetes\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.952672 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951675    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/af23cea6-b0da-4b1c-a704-d24956242e46-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kv86r\" (UID: \"af23cea6-b0da-4b1c-a704-d24956242e46\") " pod="openshift-multus/multus-additional-cni-plugins-kv86r"
Sep 18 13:15:06.952672 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951693    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-etc-sysctl-conf\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.952672 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951718    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-etc-systemd\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.952672 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951740    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-host-run-netns\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.952672 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951750    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/566608dd-e846-4eec-9f7a-9596f671090e-cni-binary-copy\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.953395 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951765    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-sys\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.953395 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951793    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47797f8b-a736-4caa-9b4a-6de71b38c001-metrics-certs\") pod \"network-metrics-daemon-qlzxp\" (UID: \"47797f8b-a736-4caa-9b4a-6de71b38c001\") " pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:06.953395 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951798    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/af23cea6-b0da-4b1c-a704-d24956242e46-cni-binary-copy\") pod \"multus-additional-cni-plugins-kv86r\" (UID: \"af23cea6-b0da-4b1c-a704-d24956242e46\") " pod="openshift-multus/multus-additional-cni-plugins-kv86r"
Sep 18 13:15:06.953395 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951807    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-system-cni-dir\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.953395 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951817    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgmvf\" (UniqueName: \"kubernetes.io/projected/990bd81d-70fa-437e-8c1c-2521d4403925-kube-api-access-lgmvf\") pod \"aws-ebs-csi-driver-node-4f7mc\" (UID: \"990bd81d-70fa-437e-8c1c-2521d4403925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc"
Sep 18 13:15:06.953395 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951879    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-multus-cni-dir\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.953395 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.950974    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-etc-sysctl-d\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.953395 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951941    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/af23cea6-b0da-4b1c-a704-d24956242e46-cnibin\") pod \"multus-additional-cni-plugins-kv86r\" (UID: \"af23cea6-b0da-4b1c-a704-d24956242e46\") " pod="openshift-multus/multus-additional-cni-plugins-kv86r"
Sep 18 13:15:06.953395 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951948    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/af23cea6-b0da-4b1c-a704-d24956242e46-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kv86r\" (UID: \"af23cea6-b0da-4b1c-a704-d24956242e46\") " pod="openshift-multus/multus-additional-cni-plugins-kv86r"
Sep 18 13:15:06.953395 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.951973    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af23cea6-b0da-4b1c-a704-d24956242e46-system-cni-dir\") pod \"multus-additional-cni-plugins-kv86r\" (UID: \"af23cea6-b0da-4b1c-a704-d24956242e46\") " pod="openshift-multus/multus-additional-cni-plugins-kv86r"
Sep 18 13:15:06.953395 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:06.951986    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:15:06.953395 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.952002    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/af23cea6-b0da-4b1c-a704-d24956242e46-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kv86r\" (UID: \"af23cea6-b0da-4b1c-a704-d24956242e46\") " pod="openshift-multus/multus-additional-cni-plugins-kv86r"
Sep 18 13:15:06.953395 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.952030    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-etc-sysconfig\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.953395 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:06.952063    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e310da4-ab19-4631-9717-86fb07925256-original-pull-secret podName:7e310da4-ab19-4631-9717-86fb07925256 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:07.452027284 +0000 UTC m=+4.807487106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e310da4-ab19-4631-9717-86fb07925256-original-pull-secret") pod "global-pull-secret-syncer-pc5zc" (UID: "7e310da4-ab19-4631-9717-86fb07925256") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:15:06.953395 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.952080    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-etc-sysconfig\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.953395 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.952088    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-host-var-lib-cni-multus\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.953395 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.952116    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/af23cea6-b0da-4b1c-a704-d24956242e46-system-cni-dir\") pod \"multus-additional-cni-plugins-kv86r\" (UID: \"af23cea6-b0da-4b1c-a704-d24956242e46\") " pod="openshift-multus/multus-additional-cni-plugins-kv86r"
Sep 18 13:15:06.954041 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.952148    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-multus-socket-dir-parent\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.954041 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.952199    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-etc-kubernetes\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.954041 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.952266    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-var-lib-kubelet\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.954041 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.952322    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-host-run-netns\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.954041 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.952382    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-etc-systemd\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.954041 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.952323    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-etc-sysctl-conf\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.954041 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:06.952400    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:15:06.954041 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.952441    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-cnibin\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.954041 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.952465    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-host-run-multus-certs\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.954041 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.952488    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-host\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.954041 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:06.952498    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47797f8b-a736-4caa-9b4a-6de71b38c001-metrics-certs podName:47797f8b-a736-4caa-9b4a-6de71b38c001 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:07.452480126 +0000 UTC m=+4.807939945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47797f8b-a736-4caa-9b4a-6de71b38c001-metrics-certs") pod "network-metrics-daemon-qlzxp" (UID: "47797f8b-a736-4caa-9b4a-6de71b38c001") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:15:06.954041 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.952091    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-multus-socket-dir-parent\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.954041 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.952509    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/566608dd-e846-4eec-9f7a-9596f671090e-etc-kubernetes\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:06.954041 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.952513    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/af23cea6-b0da-4b1c-a704-d24956242e46-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kv86r\" (UID: \"af23cea6-b0da-4b1c-a704-d24956242e46\") " pod="openshift-multus/multus-additional-cni-plugins-kv86r"
Sep 18 13:15:06.954041 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.952533    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/990bd81d-70fa-437e-8c1c-2521d4403925-device-dir\") pod \"aws-ebs-csi-driver-node-4f7mc\" (UID: \"990bd81d-70fa-437e-8c1c-2521d4403925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc"
Sep 18 13:15:06.954041 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.952580    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/990bd81d-70fa-437e-8c1c-2521d4403925-etc-selinux\") pod \"aws-ebs-csi-driver-node-4f7mc\" (UID: \"990bd81d-70fa-437e-8c1c-2521d4403925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc"
Sep 18 13:15:06.954041 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.952599    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/990bd81d-70fa-437e-8c1c-2521d4403925-device-dir\") pod \"aws-ebs-csi-driver-node-4f7mc\" (UID: \"990bd81d-70fa-437e-8c1c-2521d4403925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc"
Sep 18 13:15:06.955028 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.952627    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b72095e4-8ae2-46ee-83a2-77a996aa45bd-sys\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.955028 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.954612    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b72095e4-8ae2-46ee-83a2-77a996aa45bd-tmp\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.955785 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.955764    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b72095e4-8ae2-46ee-83a2-77a996aa45bd-etc-tuned\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.976805 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.976760    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgmvf\" (UniqueName: \"kubernetes.io/projected/990bd81d-70fa-437e-8c1c-2521d4403925-kube-api-access-lgmvf\") pod \"aws-ebs-csi-driver-node-4f7mc\" (UID: \"990bd81d-70fa-437e-8c1c-2521d4403925\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc"
Sep 18 13:15:06.980581 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.980563    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx7pn\" (UniqueName: \"kubernetes.io/projected/47797f8b-a736-4caa-9b4a-6de71b38c001-kube-api-access-jx7pn\") pod \"network-metrics-daemon-qlzxp\" (UID: \"47797f8b-a736-4caa-9b4a-6de71b38c001\") " pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:06.983493 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:06.983465    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:15:06.983493 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:06.983494    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:15:06.983649 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:06.983508    2586 projected.go:196] Error preparing data for projected volume kube-api-access-88wcc for pod openshift-network-diagnostics/network-check-target-tmdn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:15:06.983649 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:06.983561    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b7871d5-8610-4b4a-b099-4163ff75df5e-kube-api-access-88wcc podName:1b7871d5-8610-4b4a-b099-4163ff75df5e nodeName:}" failed. No retries permitted until 2026-09-18 13:15:07.483546799 +0000 UTC m=+4.839006601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-88wcc" (UniqueName: "kubernetes.io/projected/1b7871d5-8610-4b4a-b099-4163ff75df5e-kube-api-access-88wcc") pod "network-check-target-tmdn8" (UID: "1b7871d5-8610-4b4a-b099-4163ff75df5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:15:06.983854 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.983837    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-595h9\" (UniqueName: \"kubernetes.io/projected/b72095e4-8ae2-46ee-83a2-77a996aa45bd-kube-api-access-595h9\") pod \"tuned-jxbpd\" (UID: \"b72095e4-8ae2-46ee-83a2-77a996aa45bd\") " pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:06.986170 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.986152    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj28r\" (UniqueName: \"kubernetes.io/projected/af23cea6-b0da-4b1c-a704-d24956242e46-kube-api-access-nj28r\") pod \"multus-additional-cni-plugins-kv86r\" (UID: \"af23cea6-b0da-4b1c-a704-d24956242e46\") " pod="openshift-multus/multus-additional-cni-plugins-kv86r"
Sep 18 13:15:06.986998 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.986968    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9qpn\" (UniqueName: \"kubernetes.io/projected/73749cf1-5bc4-452a-b1a4-b3ef009e3ff3-kube-api-access-h9qpn\") pod \"node-resolver-lbfgj\" (UID: \"73749cf1-5bc4-452a-b1a4-b3ef009e3ff3\") " pod="openshift-dns/node-resolver-lbfgj"
Sep 18 13:15:06.990236 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:06.990210    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt4jz\" (UniqueName: \"kubernetes.io/projected/566608dd-e846-4eec-9f7a-9596f671090e-kube-api-access-zt4jz\") pod \"multus-vsgwg\" (UID: \"566608dd-e846-4eec-9f7a-9596f671090e\") " pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:07.037661 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:07.037589    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s99br"
Sep 18 13:15:07.046689 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:07.046668    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5td8r"
Sep 18 13:15:07.062572 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:07.062547    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:07.070730 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:07.070710    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fmxrm"
Sep 18 13:15:07.078046 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:07.078018    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc"
Sep 18 13:15:07.085770 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:07.085750    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vsgwg"
Sep 18 13:15:07.096283 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:07.096267    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lbfgj"
Sep 18 13:15:07.103470 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:07.103456    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kv86r"
Sep 18 13:15:07.109514 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:07.109489    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jxbpd"
Sep 18 13:15:07.133549 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:07.133521    2586 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 13:15:07.457440 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:07.456819    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e310da4-ab19-4631-9717-86fb07925256-original-pull-secret\") pod \"global-pull-secret-syncer-pc5zc\" (UID: \"7e310da4-ab19-4631-9717-86fb07925256\") " pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:07.457440 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:07.456881    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47797f8b-a736-4caa-9b4a-6de71b38c001-metrics-certs\") pod \"network-metrics-daemon-qlzxp\" (UID: \"47797f8b-a736-4caa-9b4a-6de71b38c001\") " pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:07.457440 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:07.457004    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:15:07.457440 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:07.457005    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:15:07.457440 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:07.457067    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47797f8b-a736-4caa-9b4a-6de71b38c001-metrics-certs podName:47797f8b-a736-4caa-9b4a-6de71b38c001 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:08.457049815 +0000 UTC m=+5.812509641 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47797f8b-a736-4caa-9b4a-6de71b38c001-metrics-certs") pod "network-metrics-daemon-qlzxp" (UID: "47797f8b-a736-4caa-9b4a-6de71b38c001") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:15:07.457440 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:07.457081    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e310da4-ab19-4631-9717-86fb07925256-original-pull-secret podName:7e310da4-ab19-4631-9717-86fb07925256 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:08.457075178 +0000 UTC m=+5.812534981 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e310da4-ab19-4631-9717-86fb07925256-original-pull-secret") pod "global-pull-secret-syncer-pc5zc" (UID: "7e310da4-ab19-4631-9717-86fb07925256") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:15:07.557438 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:07.557390    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88wcc\" (UniqueName: \"kubernetes.io/projected/1b7871d5-8610-4b4a-b099-4163ff75df5e-kube-api-access-88wcc\") pod \"network-check-target-tmdn8\" (UID: \"1b7871d5-8610-4b4a-b099-4163ff75df5e\") " pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:07.557601 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:07.557579    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:15:07.557649 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:07.557609    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:15:07.557649 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:07.557623    2586 projected.go:196] Error preparing data for projected volume kube-api-access-88wcc for pod openshift-network-diagnostics/network-check-target-tmdn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:15:07.557729 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:07.557685    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b7871d5-8610-4b4a-b099-4163ff75df5e-kube-api-access-88wcc podName:1b7871d5-8610-4b4a-b099-4163ff75df5e nodeName:}" failed. No retries permitted until 2026-09-18 13:15:08.557664713 +0000 UTC m=+5.913124532 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-88wcc" (UniqueName: "kubernetes.io/projected/1b7871d5-8610-4b4a-b099-4163ff75df5e-kube-api-access-88wcc") pod "network-check-target-tmdn8" (UID: "1b7871d5-8610-4b4a-b099-4163ff75df5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:15:07.844173 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:07.844086    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:07.844714 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:07.844235    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlzxp" podUID="47797f8b-a736-4caa-9b4a-6de71b38c001"
Sep 18 13:15:07.850142 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:07.850100    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 13:10:05 +0000 UTC" deadline="2028-06-19 16:21:12.917986926 +0000 UTC"
Sep 18 13:15:07.850225 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:07.850142    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15363h6m5.067848518s"
Sep 18 13:15:07.899484 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:07.899456    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7721f73b_5184_4404_b65b_774dfff4df76.slice/crio-5f35c5aceea3da297fa9a2059fdf2dcb6263d99698f4f778960eb3b032d118e6 WatchSource:0}: Error finding container 5f35c5aceea3da297fa9a2059fdf2dcb6263d99698f4f778960eb3b032d118e6: Status 404 returned error can't find the container with id 5f35c5aceea3da297fa9a2059fdf2dcb6263d99698f4f778960eb3b032d118e6
Sep 18 13:15:07.902629 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:07.902595    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73749cf1_5bc4_452a_b1a4_b3ef009e3ff3.slice/crio-fe49077c950758a31f2dc19aeb04f1304e598fbd331d951397456a77d88d16c5 WatchSource:0}: Error finding container fe49077c950758a31f2dc19aeb04f1304e598fbd331d951397456a77d88d16c5: Status 404 returned error can't find the container with id fe49077c950758a31f2dc19aeb04f1304e598fbd331d951397456a77d88d16c5
Sep 18 13:15:07.905092 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:07.905065    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990bd81d_70fa_437e_8c1c_2521d4403925.slice/crio-f75593612c9004f17bc63de3144fb457c065b00313659370a97d411e5d76a185 WatchSource:0}: Error finding container f75593612c9004f17bc63de3144fb457c065b00313659370a97d411e5d76a185: Status 404 returned error can't find the container with id f75593612c9004f17bc63de3144fb457c065b00313659370a97d411e5d76a185
Sep 18 13:15:07.906183 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:07.906158    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc32441_1189_4113_bf17_2769fa23a747.slice/crio-a2cebc6ac83615cbaa41dd0966d13c861eba369bfeaa873cba155c8f1187eb06 WatchSource:0}: Error finding container a2cebc6ac83615cbaa41dd0966d13c861eba369bfeaa873cba155c8f1187eb06: Status 404 returned error can't find the container with id a2cebc6ac83615cbaa41dd0966d13c861eba369bfeaa873cba155c8f1187eb06
Sep 18 13:15:07.907167 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:07.907126    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode53a6d50_eb67_499c_9626_d7788ef58cef.slice/crio-fd93285e1dc535907619160f60c6e1eebf011067efdd08b96da4e8ad9a6babd5 WatchSource:0}: Error finding container fd93285e1dc535907619160f60c6e1eebf011067efdd08b96da4e8ad9a6babd5: Status 404 returned error can't find the container with id fd93285e1dc535907619160f60c6e1eebf011067efdd08b96da4e8ad9a6babd5
Sep 18 13:15:08.464013 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:08.463719    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e310da4-ab19-4631-9717-86fb07925256-original-pull-secret\") pod \"global-pull-secret-syncer-pc5zc\" (UID: \"7e310da4-ab19-4631-9717-86fb07925256\") " pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:08.464211 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:08.464054    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47797f8b-a736-4caa-9b4a-6de71b38c001-metrics-certs\") pod \"network-metrics-daemon-qlzxp\" (UID: \"47797f8b-a736-4caa-9b4a-6de71b38c001\") " pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:08.464211 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:08.464190    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:15:08.464340 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:08.464266    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47797f8b-a736-4caa-9b4a-6de71b38c001-metrics-certs podName:47797f8b-a736-4caa-9b4a-6de71b38c001 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:10.464236754 +0000 UTC m=+7.819696570 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47797f8b-a736-4caa-9b4a-6de71b38c001-metrics-certs") pod "network-metrics-daemon-qlzxp" (UID: "47797f8b-a736-4caa-9b4a-6de71b38c001") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:15:08.465111 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:08.464996    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:15:08.465111 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:08.465091    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e310da4-ab19-4631-9717-86fb07925256-original-pull-secret podName:7e310da4-ab19-4631-9717-86fb07925256 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:10.465075343 +0000 UTC m=+7.820535150 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e310da4-ab19-4631-9717-86fb07925256-original-pull-secret") pod "global-pull-secret-syncer-pc5zc" (UID: "7e310da4-ab19-4631-9717-86fb07925256") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:15:08.565786 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:08.565037    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88wcc\" (UniqueName: \"kubernetes.io/projected/1b7871d5-8610-4b4a-b099-4163ff75df5e-kube-api-access-88wcc\") pod \"network-check-target-tmdn8\" (UID: \"1b7871d5-8610-4b4a-b099-4163ff75df5e\") " pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:08.565786 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:08.565259    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:15:08.565786 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:08.565286    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:15:08.565786 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:08.565298    2586 projected.go:196] Error preparing data for projected volume kube-api-access-88wcc for pod openshift-network-diagnostics/network-check-target-tmdn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:15:08.565786 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:08.565362    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b7871d5-8610-4b4a-b099-4163ff75df5e-kube-api-access-88wcc podName:1b7871d5-8610-4b4a-b099-4163ff75df5e nodeName:}" failed. No retries permitted until 2026-09-18 13:15:10.565343862 +0000 UTC m=+7.920803685 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-88wcc" (UniqueName: "kubernetes.io/projected/1b7871d5-8610-4b4a-b099-4163ff75df5e-kube-api-access-88wcc") pod "network-check-target-tmdn8" (UID: "1b7871d5-8610-4b4a-b099-4163ff75df5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:15:08.845587 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:08.845027    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:08.845587 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:08.845160    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmdn8" podUID="1b7871d5-8610-4b4a-b099-4163ff75df5e"
Sep 18 13:15:08.846616 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:08.846470    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:08.846616 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:08.846573    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pc5zc" podUID="7e310da4-ab19-4631-9717-86fb07925256"
Sep 18 13:15:08.864474 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:08.864356    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal" event={"ID":"31c0bde74e3407e1ddada35fdb2a70e7","Type":"ContainerDied","Data":"155eeaea6ff60c461887f0b14fdc4c2256f52c50c9ea7c197c9a50d7ab7a5246"}
Sep 18 13:15:08.864717 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:08.864622    2586 generic.go:356] "Generic (PLEG): container finished" podID="31c0bde74e3407e1ddada35fdb2a70e7" containerID="155eeaea6ff60c461887f0b14fdc4c2256f52c50c9ea7c197c9a50d7ab7a5246" exitCode=0
Sep 18 13:15:08.877283 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:08.877222    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kv86r" event={"ID":"af23cea6-b0da-4b1c-a704-d24956242e46","Type":"ContainerStarted","Data":"d481bcaec02ee84b6676c18f6f48d5f9829a3d5ea7ec8ed12c2602028584f17e"}
Sep 18 13:15:08.888704 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:08.888660    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vsgwg" event={"ID":"566608dd-e846-4eec-9f7a-9596f671090e","Type":"ContainerStarted","Data":"66c6f199905a8bbadbd817e4cf51651eb111189b016d9a494d56f0e9897e2cae"}
Sep 18 13:15:08.902358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:08.902323    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw" event={"ID":"1cc32441-1189-4113-bf17-2769fa23a747","Type":"ContainerStarted","Data":"a2cebc6ac83615cbaa41dd0966d13c861eba369bfeaa873cba155c8f1187eb06"}
Sep 18 13:15:08.905794 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:08.905763    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s99br" event={"ID":"7721f73b-5184-4404-b65b-774dfff4df76","Type":"ContainerStarted","Data":"5f35c5aceea3da297fa9a2059fdf2dcb6263d99698f4f778960eb3b032d118e6"}
Sep 18 13:15:08.922380 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:08.922344    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-221.ec2.internal" event={"ID":"523614729ae9b688582b641ea2fafde6","Type":"ContainerStarted","Data":"4db497022d247ab2d43a50c49c83c9cb47ad53b1f275b3fcfad06f9f02e6f141"}
Sep 18 13:15:08.938336 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:08.938303    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jxbpd" event={"ID":"b72095e4-8ae2-46ee-83a2-77a996aa45bd","Type":"ContainerStarted","Data":"b10b2e0bf75e3554512c7dc69d7b86f338b317517736b12734afd3204240e4eb"}
Sep 18 13:15:08.950090 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:08.950060    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5td8r" event={"ID":"b6e6324a-7c14-4edb-9e82-0729e23ce5b8","Type":"ContainerStarted","Data":"d05e0996b7edfeeba848d89f690ea4146388d296dc576148478d0b2bf9141093"}
Sep 18 13:15:08.960661 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:08.960628    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fmxrm" event={"ID":"e53a6d50-eb67-499c-9626-d7788ef58cef","Type":"ContainerStarted","Data":"fd93285e1dc535907619160f60c6e1eebf011067efdd08b96da4e8ad9a6babd5"}
Sep 18 13:15:08.969282 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:08.969253    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc" event={"ID":"990bd81d-70fa-437e-8c1c-2521d4403925","Type":"ContainerStarted","Data":"f75593612c9004f17bc63de3144fb457c065b00313659370a97d411e5d76a185"}
Sep 18 13:15:08.982705 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:08.982673    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lbfgj" event={"ID":"73749cf1-5bc4-452a-b1a4-b3ef009e3ff3","Type":"ContainerStarted","Data":"fe49077c950758a31f2dc19aeb04f1304e598fbd331d951397456a77d88d16c5"}
Sep 18 13:15:09.844157 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:09.843922    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:09.844157 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:09.844075    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlzxp" podUID="47797f8b-a736-4caa-9b4a-6de71b38c001"
Sep 18 13:15:10.006595 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:10.006514    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal" event={"ID":"31c0bde74e3407e1ddada35fdb2a70e7","Type":"ContainerStarted","Data":"a31157fd4a6d8da2d7dd2a99a69e19c8e75f1060c944871a332cf9cbbb9cb367"}
Sep 18 13:15:10.024973 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:10.024615    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-221.ec2.internal" podStartSLOduration=5.024591031 podStartE2EDuration="5.024591031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:08.944091661 +0000 UTC m=+6.299551487" watchObservedRunningTime="2026-09-18 13:15:10.024591031 +0000 UTC m=+7.380050858"
Sep 18 13:15:10.024973 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:10.024811    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-221.ec2.internal" podStartSLOduration=5.024797735 podStartE2EDuration="5.024797735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:10.023153534 +0000 UTC m=+7.378613386" watchObservedRunningTime="2026-09-18 13:15:10.024797735 +0000 UTC m=+7.380257562"
Sep 18 13:15:10.481551 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:10.481516    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e310da4-ab19-4631-9717-86fb07925256-original-pull-secret\") pod \"global-pull-secret-syncer-pc5zc\" (UID: \"7e310da4-ab19-4631-9717-86fb07925256\") " pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:10.481751 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:10.481566    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47797f8b-a736-4caa-9b4a-6de71b38c001-metrics-certs\") pod \"network-metrics-daemon-qlzxp\" (UID: \"47797f8b-a736-4caa-9b4a-6de71b38c001\") " pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:10.481751 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:10.481680    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:15:10.481751 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:10.481727    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47797f8b-a736-4caa-9b4a-6de71b38c001-metrics-certs podName:47797f8b-a736-4caa-9b4a-6de71b38c001 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:14.481715184 +0000 UTC m=+11.837174987 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47797f8b-a736-4caa-9b4a-6de71b38c001-metrics-certs") pod "network-metrics-daemon-qlzxp" (UID: "47797f8b-a736-4caa-9b4a-6de71b38c001") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:15:10.481941 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:10.481799    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:15:10.481941 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:10.481831    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e310da4-ab19-4631-9717-86fb07925256-original-pull-secret podName:7e310da4-ab19-4631-9717-86fb07925256 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:14.481824268 +0000 UTC m=+11.837284071 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e310da4-ab19-4631-9717-86fb07925256-original-pull-secret") pod "global-pull-secret-syncer-pc5zc" (UID: "7e310da4-ab19-4631-9717-86fb07925256") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:15:10.582982 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:10.582942    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88wcc\" (UniqueName: \"kubernetes.io/projected/1b7871d5-8610-4b4a-b099-4163ff75df5e-kube-api-access-88wcc\") pod \"network-check-target-tmdn8\" (UID: \"1b7871d5-8610-4b4a-b099-4163ff75df5e\") " pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:10.583193 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:10.583172    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:15:10.583262 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:10.583205    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:15:10.583262 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:10.583217    2586 projected.go:196] Error preparing data for projected volume kube-api-access-88wcc for pod openshift-network-diagnostics/network-check-target-tmdn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:15:10.583383 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:10.583276    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b7871d5-8610-4b4a-b099-4163ff75df5e-kube-api-access-88wcc podName:1b7871d5-8610-4b4a-b099-4163ff75df5e nodeName:}" failed. No retries permitted until 2026-09-18 13:15:14.583258071 +0000 UTC m=+11.938717876 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-88wcc" (UniqueName: "kubernetes.io/projected/1b7871d5-8610-4b4a-b099-4163ff75df5e-kube-api-access-88wcc") pod "network-check-target-tmdn8" (UID: "1b7871d5-8610-4b4a-b099-4163ff75df5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:15:10.853119 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:10.853039    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:10.853286 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:10.853225    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmdn8" podUID="1b7871d5-8610-4b4a-b099-4163ff75df5e"
Sep 18 13:15:10.853754 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:10.853708    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:10.853877 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:10.853855    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pc5zc" podUID="7e310da4-ab19-4631-9717-86fb07925256"
Sep 18 13:15:11.843943 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:11.843892    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:11.844426 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:11.844060    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlzxp" podUID="47797f8b-a736-4caa-9b4a-6de71b38c001"
Sep 18 13:15:12.843757 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:12.843725    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:12.843757 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:12.843749    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:12.844023 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:12.843859    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmdn8" podUID="1b7871d5-8610-4b4a-b099-4163ff75df5e"
Sep 18 13:15:12.844483 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:12.844013    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pc5zc" podUID="7e310da4-ab19-4631-9717-86fb07925256"
Sep 18 13:15:13.844000 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:13.843966    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:13.844215 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:13.844160    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlzxp" podUID="47797f8b-a736-4caa-9b4a-6de71b38c001"
Sep 18 13:15:14.519594 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:14.519555    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47797f8b-a736-4caa-9b4a-6de71b38c001-metrics-certs\") pod \"network-metrics-daemon-qlzxp\" (UID: \"47797f8b-a736-4caa-9b4a-6de71b38c001\") " pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:14.519767 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:14.519639    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e310da4-ab19-4631-9717-86fb07925256-original-pull-secret\") pod \"global-pull-secret-syncer-pc5zc\" (UID: \"7e310da4-ab19-4631-9717-86fb07925256\") " pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:14.519767 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:14.519757    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:15:14.519882 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:14.519819    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e310da4-ab19-4631-9717-86fb07925256-original-pull-secret podName:7e310da4-ab19-4631-9717-86fb07925256 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:22.519801425 +0000 UTC m=+19.875261243 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e310da4-ab19-4631-9717-86fb07925256-original-pull-secret") pod "global-pull-secret-syncer-pc5zc" (UID: "7e310da4-ab19-4631-9717-86fb07925256") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:15:14.520282 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:14.520244    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:15:14.520390 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:14.520300    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47797f8b-a736-4caa-9b4a-6de71b38c001-metrics-certs podName:47797f8b-a736-4caa-9b4a-6de71b38c001 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:22.520283953 +0000 UTC m=+19.875743770 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47797f8b-a736-4caa-9b4a-6de71b38c001-metrics-certs") pod "network-metrics-daemon-qlzxp" (UID: "47797f8b-a736-4caa-9b4a-6de71b38c001") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:15:14.620463 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:14.620417    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88wcc\" (UniqueName: \"kubernetes.io/projected/1b7871d5-8610-4b4a-b099-4163ff75df5e-kube-api-access-88wcc\") pod \"network-check-target-tmdn8\" (UID: \"1b7871d5-8610-4b4a-b099-4163ff75df5e\") " pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:14.620650 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:14.620637    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:15:14.620717 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:14.620658    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:15:14.620717 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:14.620680    2586 projected.go:196] Error preparing data for projected volume kube-api-access-88wcc for pod openshift-network-diagnostics/network-check-target-tmdn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:15:14.620801 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:14.620736    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b7871d5-8610-4b4a-b099-4163ff75df5e-kube-api-access-88wcc podName:1b7871d5-8610-4b4a-b099-4163ff75df5e nodeName:}" failed. No retries permitted until 2026-09-18 13:15:22.620719796 +0000 UTC m=+19.976179613 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-88wcc" (UniqueName: "kubernetes.io/projected/1b7871d5-8610-4b4a-b099-4163ff75df5e-kube-api-access-88wcc") pod "network-check-target-tmdn8" (UID: "1b7871d5-8610-4b4a-b099-4163ff75df5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:15:14.847279 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:14.847206    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:14.847702 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:14.847207    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:14.847702 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:14.847327    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pc5zc" podUID="7e310da4-ab19-4631-9717-86fb07925256"
Sep 18 13:15:14.847702 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:14.847430    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmdn8" podUID="1b7871d5-8610-4b4a-b099-4163ff75df5e"
Sep 18 13:15:15.843827 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:15.843401    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:15.843827 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:15.843519    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlzxp" podUID="47797f8b-a736-4caa-9b4a-6de71b38c001"
Sep 18 13:15:16.846589 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:16.846547    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:16.847034 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:16.846659    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pc5zc" podUID="7e310da4-ab19-4631-9717-86fb07925256"
Sep 18 13:15:16.847034 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:16.847015    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:16.847142 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:16.847085    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmdn8" podUID="1b7871d5-8610-4b4a-b099-4163ff75df5e"
Sep 18 13:15:17.844083 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:17.844048    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:17.844263 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:17.844182    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlzxp" podUID="47797f8b-a736-4caa-9b4a-6de71b38c001"
Sep 18 13:15:18.843489 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:18.843452    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:18.843924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:18.843450    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:18.843924 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:18.843579    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmdn8" podUID="1b7871d5-8610-4b4a-b099-4163ff75df5e"
Sep 18 13:15:18.843924 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:18.843647    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pc5zc" podUID="7e310da4-ab19-4631-9717-86fb07925256"
Sep 18 13:15:19.843660 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:19.843622    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:19.844065 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:19.843768    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlzxp" podUID="47797f8b-a736-4caa-9b4a-6de71b38c001"
Sep 18 13:15:20.844266 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:20.844196    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:20.844718 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:20.844277    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:20.844718 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:20.844361    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmdn8" podUID="1b7871d5-8610-4b4a-b099-4163ff75df5e"
Sep 18 13:15:20.844718 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:20.844498    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pc5zc" podUID="7e310da4-ab19-4631-9717-86fb07925256"
Sep 18 13:15:21.844056 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:21.844023    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:21.844209 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:21.844137    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlzxp" podUID="47797f8b-a736-4caa-9b4a-6de71b38c001"
Sep 18 13:15:22.581045 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:22.581008    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47797f8b-a736-4caa-9b4a-6de71b38c001-metrics-certs\") pod \"network-metrics-daemon-qlzxp\" (UID: \"47797f8b-a736-4caa-9b4a-6de71b38c001\") " pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:22.581532 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:22.581076    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e310da4-ab19-4631-9717-86fb07925256-original-pull-secret\") pod \"global-pull-secret-syncer-pc5zc\" (UID: \"7e310da4-ab19-4631-9717-86fb07925256\") " pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:22.581532 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:22.581182    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:15:22.581532 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:22.581185    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:15:22.581532 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:22.581240    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e310da4-ab19-4631-9717-86fb07925256-original-pull-secret podName:7e310da4-ab19-4631-9717-86fb07925256 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:38.581226165 +0000 UTC m=+35.936685969 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e310da4-ab19-4631-9717-86fb07925256-original-pull-secret") pod "global-pull-secret-syncer-pc5zc" (UID: "7e310da4-ab19-4631-9717-86fb07925256") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:15:22.581532 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:22.581257    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47797f8b-a736-4caa-9b4a-6de71b38c001-metrics-certs podName:47797f8b-a736-4caa-9b4a-6de71b38c001 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:38.581248082 +0000 UTC m=+35.936707885 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47797f8b-a736-4caa-9b4a-6de71b38c001-metrics-certs") pod "network-metrics-daemon-qlzxp" (UID: "47797f8b-a736-4caa-9b4a-6de71b38c001") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:15:22.681827 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:22.681791    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88wcc\" (UniqueName: \"kubernetes.io/projected/1b7871d5-8610-4b4a-b099-4163ff75df5e-kube-api-access-88wcc\") pod \"network-check-target-tmdn8\" (UID: \"1b7871d5-8610-4b4a-b099-4163ff75df5e\") " pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:22.682017 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:22.681996    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:15:22.682095 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:22.682025    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:15:22.682095 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:22.682038    2586 projected.go:196] Error preparing data for projected volume kube-api-access-88wcc for pod openshift-network-diagnostics/network-check-target-tmdn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:15:22.682191 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:22.682101    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b7871d5-8610-4b4a-b099-4163ff75df5e-kube-api-access-88wcc podName:1b7871d5-8610-4b4a-b099-4163ff75df5e nodeName:}" failed. No retries permitted until 2026-09-18 13:15:38.682083367 +0000 UTC m=+36.037543185 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-88wcc" (UniqueName: "kubernetes.io/projected/1b7871d5-8610-4b4a-b099-4163ff75df5e-kube-api-access-88wcc") pod "network-check-target-tmdn8" (UID: "1b7871d5-8610-4b4a-b099-4163ff75df5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:15:22.844088 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:22.844005    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:22.844226 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:22.844011    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:22.844226 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:22.844145    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmdn8" podUID="1b7871d5-8610-4b4a-b099-4163ff75df5e"
Sep 18 13:15:22.844226 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:22.844203    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pc5zc" podUID="7e310da4-ab19-4631-9717-86fb07925256"
Sep 18 13:15:23.843884 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:23.843851    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:23.844369 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:23.844012    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlzxp" podUID="47797f8b-a736-4caa-9b4a-6de71b38c001"
Sep 18 13:15:24.844126 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:24.844082    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:24.844601 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:24.844283    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmdn8" podUID="1b7871d5-8610-4b4a-b099-4163ff75df5e"
Sep 18 13:15:24.844694 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:24.844673    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:24.844794 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:24.844777    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pc5zc" podUID="7e310da4-ab19-4631-9717-86fb07925256"
Sep 18 13:15:25.844375 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:25.844215    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:25.844712 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:25.844458    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlzxp" podUID="47797f8b-a736-4caa-9b4a-6de71b38c001"
Sep 18 13:15:26.042313 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:26.042276    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lbfgj" event={"ID":"73749cf1-5bc4-452a-b1a4-b3ef009e3ff3","Type":"ContainerStarted","Data":"65ca4717b8ff856f8d14639c2c20d5896cd7fe1a9dc230c270765bac92829a2a"}
Sep 18 13:15:26.043979 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:26.043951    2586 generic.go:356] "Generic (PLEG): container finished" podID="af23cea6-b0da-4b1c-a704-d24956242e46" containerID="4c656b6ef2bf2d73aa77cb81229e93b276bcca301892792ebe0366d967b61fa2" exitCode=0
Sep 18 13:15:26.044096 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:26.044042    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kv86r" event={"ID":"af23cea6-b0da-4b1c-a704-d24956242e46","Type":"ContainerDied","Data":"4c656b6ef2bf2d73aa77cb81229e93b276bcca301892792ebe0366d967b61fa2"}
Sep 18 13:15:26.047666 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:26.046105    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vsgwg" event={"ID":"566608dd-e846-4eec-9f7a-9596f671090e","Type":"ContainerStarted","Data":"c5987621856f40a6a11812ce6b5fad39f47b5929d19102e33b5d34adfaf053e2"}
Sep 18 13:15:26.048275 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:26.048119    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lcqcw_1cc32441-1189-4113-bf17-2769fa23a747/ovn-acl-logging/0.log"
Sep 18 13:15:26.048510 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:26.048485    2586 generic.go:356] "Generic (PLEG): container finished" podID="1cc32441-1189-4113-bf17-2769fa23a747" containerID="f901c9d0683798aa56e7f8f5f21941951d4921793982ca66aae1b540630e3cc3" exitCode=1
Sep 18 13:15:26.048581 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:26.048563    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw" event={"ID":"1cc32441-1189-4113-bf17-2769fa23a747","Type":"ContainerStarted","Data":"17556e9797a2804c991e34e026566f9b00649debf3bf2fae53889e0aa9dbaa51"}
Sep 18 13:15:26.048654 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:26.048635    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw" event={"ID":"1cc32441-1189-4113-bf17-2769fa23a747","Type":"ContainerDied","Data":"f901c9d0683798aa56e7f8f5f21941951d4921793982ca66aae1b540630e3cc3"}
Sep 18 13:15:26.048702 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:26.048684    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw" event={"ID":"1cc32441-1189-4113-bf17-2769fa23a747","Type":"ContainerStarted","Data":"dfc70c9089731d6f0a7325b05e9249f6df6051bcbd0ddfea76f29f07462f2745"}
Sep 18 13:15:26.050195 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:26.050175    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s99br" event={"ID":"7721f73b-5184-4404-b65b-774dfff4df76","Type":"ContainerStarted","Data":"cbf73f533153186bb7c379e5dfc8546e91186032f8f937638342cb128832ad9d"}
Sep 18 13:15:26.052289 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:26.052026    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jxbpd" event={"ID":"b72095e4-8ae2-46ee-83a2-77a996aa45bd","Type":"ContainerStarted","Data":"8829f9b128e0f6674c282bbfc61f85b2e00b96d7e17d76b8fea9abbd6a3dbdd6"}
Sep 18 13:15:26.057519 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:26.057482    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fmxrm" event={"ID":"e53a6d50-eb67-499c-9626-d7788ef58cef","Type":"ContainerStarted","Data":"aba6b9d2ff84f2ebd8e29be77d0ed90cd3a2205c6079344c7ca3f513883521fb"}
Sep 18 13:15:26.058994 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:26.058969    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc" event={"ID":"990bd81d-70fa-437e-8c1c-2521d4403925","Type":"ContainerStarted","Data":"1a1dd58bfb4c11537568f388d43b76e4f6baca846568bf0b610818822adb75b9"}
Sep 18 13:15:26.073879 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:26.073794    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lbfgj" podStartSLOduration=3.495457243 podStartE2EDuration="21.073775235s" totalImagesPullingTime="17.578317992s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:26.073316112 +0000 UTC m=+23.428775936" watchObservedRunningTime="2026-09-18 13:15:26.073775235 +0000 UTC m=+23.429235061"
Sep 18 13:15:26.149361 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:26.149127    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fmxrm" podStartSLOduration=4.644432835 podStartE2EDuration="22.1491084s" totalImagesPullingTime="17.504675565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:26.112798035 +0000 UTC m=+23.468257863" watchObservedRunningTime="2026-09-18 13:15:26.1491084 +0000 UTC m=+23.504568228"
Sep 18 13:15:26.224202 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:26.224159    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jxbpd" podStartSLOduration=3.651167753 podStartE2EDuration="21.224142641s" totalImagesPullingTime="17.572974888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:26.182384879 +0000 UTC m=+23.537844704" watchObservedRunningTime="2026-09-18 13:15:26.224142641 +0000 UTC m=+23.579602448"
Sep 18 13:15:26.247849 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:26.247805    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vsgwg" podStartSLOduration=4.656070303 podStartE2EDuration="22.247787999s" totalImagesPullingTime="17.591717696s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:26.226168926 +0000 UTC m=+23.581628752" watchObservedRunningTime="2026-09-18 13:15:26.247787999 +0000 UTC m=+23.603247894"
Sep 18 13:15:26.591196 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:26.591169    2586 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 13:15:26.825884 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:26.825727    2586 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T13:15:26.591193033Z","UUID":"e853e160-b93e-4178-a5f9-38bbff8459b2","Handler":null,"Name":"","Endpoint":""}
Sep 18 13:15:26.827154 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:26.827130    2586 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 13:15:26.827254 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:26.827162    2586 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 13:15:26.843519 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:26.843496    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:26.843625 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:26.843500    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:26.843625 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:26.843600    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmdn8" podUID="1b7871d5-8610-4b4a-b099-4163ff75df5e"
Sep 18 13:15:26.843702 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:26.843661    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pc5zc" podUID="7e310da4-ab19-4631-9717-86fb07925256"
Sep 18 13:15:27.065134 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:27.065102    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lcqcw_1cc32441-1189-4113-bf17-2769fa23a747/ovn-acl-logging/0.log"
Sep 18 13:15:27.066261 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:27.066101    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw" event={"ID":"1cc32441-1189-4113-bf17-2769fa23a747","Type":"ContainerStarted","Data":"6a80671cec9ba17450ebf702baf75253f0d27e84ce071e32b3b02c8034c73ed1"}
Sep 18 13:15:27.066261 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:27.066182    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw" event={"ID":"1cc32441-1189-4113-bf17-2769fa23a747","Type":"ContainerStarted","Data":"1fff48e9567a78eedc82f4234b85c8c0e85dadd3654de827b6f75422d4e0c83f"}
Sep 18 13:15:27.066261 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:27.066213    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw" event={"ID":"1cc32441-1189-4113-bf17-2769fa23a747","Type":"ContainerStarted","Data":"c900184a2105617f877b89954e7a3f79e2bdc742c84a612f450132dbbb4dbf5a"}
Sep 18 13:15:27.068603 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:27.068553    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5td8r" event={"ID":"b6e6324a-7c14-4edb-9e82-0729e23ce5b8","Type":"ContainerStarted","Data":"a18e9d36779a68ba5d72bd204f67c6d817c2ca97c4ea55afe00846b8d22c7870"}
Sep 18 13:15:27.070940 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:27.070913    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc" event={"ID":"990bd81d-70fa-437e-8c1c-2521d4403925","Type":"ContainerStarted","Data":"a8555a276214a2b0d3bd693dd1bb6817ec3bdde15b8835941da2d51a618761f7"}
Sep 18 13:15:27.099043 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:27.098920    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-s99br" podStartSLOduration=5.51703029 podStartE2EDuration="23.098882577s" totalImagesPullingTime="17.581852287s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:26.247874413 +0000 UTC m=+23.603334265" watchObservedRunningTime="2026-09-18 13:15:27.098882577 +0000 UTC m=+24.454342404"
Sep 18 13:15:27.843847 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:27.843807    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:27.844011 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:27.843977    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlzxp" podUID="47797f8b-a736-4caa-9b4a-6de71b38c001"
Sep 18 13:15:28.076380 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:28.076288    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc" event={"ID":"990bd81d-70fa-437e-8c1c-2521d4403925","Type":"ContainerStarted","Data":"3e1801df94cba39aee8f8d157b250855a7ffc908cf1618945a973f23b68d8e66"}
Sep 18 13:15:28.112839 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:28.112795    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5td8r" podStartSLOduration=6.534168908 podStartE2EDuration="24.112779887s" totalImagesPullingTime="17.578610979s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:27.099118755 +0000 UTC m=+24.454578581" watchObservedRunningTime="2026-09-18 13:15:28.112779887 +0000 UTC m=+25.468239693"
Sep 18 13:15:28.412092 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:28.412069    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fmxrm"
Sep 18 13:15:28.412630 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:28.412606    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fmxrm"
Sep 18 13:15:28.450470 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:28.450420    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4f7mc" podStartSLOduration=4.900652511 podStartE2EDuration="24.450400865s" totalImagesPullingTime="19.549748354s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:15:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:28.114577427 +0000 UTC m=+25.470037257" watchObservedRunningTime="2026-09-18 13:15:28.450400865 +0000 UTC m=+25.805860680"
Sep 18 13:15:28.843985 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:28.843954    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:28.844142 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:28.844063    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmdn8" podUID="1b7871d5-8610-4b4a-b099-4163ff75df5e"
Sep 18 13:15:28.844142 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:28.844124    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:28.844242 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:28.844226    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pc5zc" podUID="7e310da4-ab19-4631-9717-86fb07925256"
Sep 18 13:15:29.082610 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:29.082577    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lcqcw_1cc32441-1189-4113-bf17-2769fa23a747/ovn-acl-logging/0.log"
Sep 18 13:15:29.083124 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:29.083096    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw" event={"ID":"1cc32441-1189-4113-bf17-2769fa23a747","Type":"ContainerStarted","Data":"4c63d0825be120aacf698a56ad3b5a2e1ae8d0256e7e0a450cea31f154eced9c"}
Sep 18 13:15:29.083456 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:29.083433    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fmxrm"
Sep 18 13:15:29.083866 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:29.083849    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fmxrm"
Sep 18 13:15:29.843716 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:29.843685    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:29.843917 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:29.843811    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlzxp" podUID="47797f8b-a736-4caa-9b4a-6de71b38c001"
Sep 18 13:15:30.843594 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:30.843404    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:30.844332 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:30.843488    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:30.844332 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:30.843662    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmdn8" podUID="1b7871d5-8610-4b4a-b099-4163ff75df5e"
Sep 18 13:15:30.844332 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:30.843785    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pc5zc" podUID="7e310da4-ab19-4631-9717-86fb07925256"
Sep 18 13:15:31.091184 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:31.091158    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lcqcw_1cc32441-1189-4113-bf17-2769fa23a747/ovn-acl-logging/0.log"
Sep 18 13:15:31.091539 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:31.091514    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw" event={"ID":"1cc32441-1189-4113-bf17-2769fa23a747","Type":"ContainerStarted","Data":"0bd1c606f7506306c40eccc422206904e46773728388a71aab8e81e2ba334a89"}
Sep 18 13:15:31.091768 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:31.091747    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:31.091840 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:31.091829    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:31.091977 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:31.091959    2586 scope.go:117] "RemoveContainer" containerID="f901c9d0683798aa56e7f8f5f21941951d4921793982ca66aae1b540630e3cc3"
Sep 18 13:15:31.093495 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:31.093473    2586 generic.go:356] "Generic (PLEG): container finished" podID="af23cea6-b0da-4b1c-a704-d24956242e46" containerID="47df2c6cf8fb16e3b1533e6e5d6f37d9fff26b3530e9c50500a189751de779ea" exitCode=0
Sep 18 13:15:31.093590 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:31.093557    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kv86r" event={"ID":"af23cea6-b0da-4b1c-a704-d24956242e46","Type":"ContainerDied","Data":"47df2c6cf8fb16e3b1533e6e5d6f37d9fff26b3530e9c50500a189751de779ea"}
Sep 18 13:15:31.108341 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:31.108320    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:31.843314 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:31.843280    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:31.843494 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:31.843395    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlzxp" podUID="47797f8b-a736-4caa-9b4a-6de71b38c001"
Sep 18 13:15:32.099267 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:32.099199    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lcqcw_1cc32441-1189-4113-bf17-2769fa23a747/ovn-acl-logging/0.log"
Sep 18 13:15:32.099760 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:32.099536    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw" event={"ID":"1cc32441-1189-4113-bf17-2769fa23a747","Type":"ContainerStarted","Data":"6dc446f7ff3d0f0e7d56573e17e41f8560bbbf86c57fa091332f00711280a6bb"}
Sep 18 13:15:32.099760 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:32.099699    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:32.116070 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:32.116038    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:15:32.149131 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:32.149073    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw" podStartSLOduration=10.309418377 podStartE2EDuration="28.149057593s" totalImagesPullingTime="17.839639216s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:32.148719426 +0000 UTC m=+29.504179253" watchObservedRunningTime="2026-09-18 13:15:32.149057593 +0000 UTC m=+29.504517417"
Sep 18 13:15:32.830420 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:32.830253    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pc5zc"]
Sep 18 13:15:32.830585 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:32.830498    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:32.830630 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:32.830576    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pc5zc" podUID="7e310da4-ab19-4631-9717-86fb07925256"
Sep 18 13:15:32.833969 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:32.833938    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qlzxp"]
Sep 18 13:15:32.834075 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:32.834061    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:32.834193 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:32.834172    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlzxp" podUID="47797f8b-a736-4caa-9b4a-6de71b38c001"
Sep 18 13:15:32.837790 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:32.837767    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tmdn8"]
Sep 18 13:15:32.837898 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:32.837879    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:32.838020 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:32.837996    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmdn8" podUID="1b7871d5-8610-4b4a-b099-4163ff75df5e"
Sep 18 13:15:33.103816 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:33.103737    2586 generic.go:356] "Generic (PLEG): container finished" podID="af23cea6-b0da-4b1c-a704-d24956242e46" containerID="2c8a9d98532761c186dc1c08a354f87e1e05114ee7986f63793f6ce8178453fe" exitCode=0
Sep 18 13:15:33.104203 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:33.103816    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kv86r" event={"ID":"af23cea6-b0da-4b1c-a704-d24956242e46","Type":"ContainerDied","Data":"2c8a9d98532761c186dc1c08a354f87e1e05114ee7986f63793f6ce8178453fe"}
Sep 18 13:15:34.845185 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:34.845103    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:34.845640 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:34.845208    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlzxp" podUID="47797f8b-a736-4caa-9b4a-6de71b38c001"
Sep 18 13:15:34.845640 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:34.845240    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:34.845640 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:34.845325    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pc5zc" podUID="7e310da4-ab19-4631-9717-86fb07925256"
Sep 18 13:15:34.845640 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:34.845349    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:34.845640 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:34.845423    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmdn8" podUID="1b7871d5-8610-4b4a-b099-4163ff75df5e"
Sep 18 13:15:35.110200 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:35.110121    2586 generic.go:356] "Generic (PLEG): container finished" podID="af23cea6-b0da-4b1c-a704-d24956242e46" containerID="88c42cdb29cc10908edddc28bfa3a461edd39edc6731df3643d553e9538bbab3" exitCode=0
Sep 18 13:15:35.110200 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:35.110183    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kv86r" event={"ID":"af23cea6-b0da-4b1c-a704-d24956242e46","Type":"ContainerDied","Data":"88c42cdb29cc10908edddc28bfa3a461edd39edc6731df3643d553e9538bbab3"}
Sep 18 13:15:36.843444 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:36.843411    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:36.843951 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:36.843524    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pc5zc" podUID="7e310da4-ab19-4631-9717-86fb07925256"
Sep 18 13:15:36.843951 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:36.843411    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:36.843951 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:36.843593    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmdn8" podUID="1b7871d5-8610-4b4a-b099-4163ff75df5e"
Sep 18 13:15:36.843951 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:36.843411    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:36.843951 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:36.843659    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlzxp" podUID="47797f8b-a736-4caa-9b4a-6de71b38c001"
Sep 18 13:15:38.604099 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:38.604064    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47797f8b-a736-4caa-9b4a-6de71b38c001-metrics-certs\") pod \"network-metrics-daemon-qlzxp\" (UID: \"47797f8b-a736-4caa-9b4a-6de71b38c001\") " pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:38.604578 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:38.604132    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e310da4-ab19-4631-9717-86fb07925256-original-pull-secret\") pod \"global-pull-secret-syncer-pc5zc\" (UID: \"7e310da4-ab19-4631-9717-86fb07925256\") " pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:38.604578 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:38.604221    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 13:15:38.604578 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:38.604221    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:15:38.604578 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:38.604285    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47797f8b-a736-4caa-9b4a-6de71b38c001-metrics-certs podName:47797f8b-a736-4caa-9b4a-6de71b38c001 nodeName:}" failed. No retries permitted until 2026-09-18 13:16:10.604267429 +0000 UTC m=+67.959727236 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/47797f8b-a736-4caa-9b4a-6de71b38c001-metrics-certs") pod "network-metrics-daemon-qlzxp" (UID: "47797f8b-a736-4caa-9b4a-6de71b38c001") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 13:15:38.604578 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:38.604302    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e310da4-ab19-4631-9717-86fb07925256-original-pull-secret podName:7e310da4-ab19-4631-9717-86fb07925256 nodeName:}" failed. No retries permitted until 2026-09-18 13:16:10.604293682 +0000 UTC m=+67.959753488 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7e310da4-ab19-4631-9717-86fb07925256-original-pull-secret") pod "global-pull-secret-syncer-pc5zc" (UID: "7e310da4-ab19-4631-9717-86fb07925256") : object "kube-system"/"original-pull-secret" not registered
Sep 18 13:15:38.705083 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:38.705049    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88wcc\" (UniqueName: \"kubernetes.io/projected/1b7871d5-8610-4b4a-b099-4163ff75df5e-kube-api-access-88wcc\") pod \"network-check-target-tmdn8\" (UID: \"1b7871d5-8610-4b4a-b099-4163ff75df5e\") " pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:38.705271 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:38.705201    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 13:15:38.705271 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:38.705222    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 13:15:38.705271 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:38.705235    2586 projected.go:196] Error preparing data for projected volume kube-api-access-88wcc for pod openshift-network-diagnostics/network-check-target-tmdn8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:15:38.705426 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:38.705293    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b7871d5-8610-4b4a-b099-4163ff75df5e-kube-api-access-88wcc podName:1b7871d5-8610-4b4a-b099-4163ff75df5e nodeName:}" failed. No retries permitted until 2026-09-18 13:16:10.705279791 +0000 UTC m=+68.060739594 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-88wcc" (UniqueName: "kubernetes.io/projected/1b7871d5-8610-4b4a-b099-4163ff75df5e-kube-api-access-88wcc") pod "network-check-target-tmdn8" (UID: "1b7871d5-8610-4b4a-b099-4163ff75df5e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 13:15:38.747432 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:38.747405    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lbfgj_73749cf1-5bc4-452a-b1a4-b3ef009e3ff3/dns-node-resolver/0.log"
Sep 18 13:15:38.843956 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:38.843866    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:38.843956 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:38.843898    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:38.844141 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:38.843866    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:38.844141 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:38.844018    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmdn8" podUID="1b7871d5-8610-4b4a-b099-4163ff75df5e"
Sep 18 13:15:38.844141 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:38.844095    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pc5zc" podUID="7e310da4-ab19-4631-9717-86fb07925256"
Sep 18 13:15:38.844256 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:38.844153    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qlzxp" podUID="47797f8b-a736-4caa-9b4a-6de71b38c001"
Sep 18 13:15:39.494804 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.494730    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-221.ec2.internal" event="NodeReady"
Sep 18 13:15:39.494971 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.494873    2586 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 13:15:39.616994 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.616963    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hp46h"]
Sep 18 13:15:39.644848 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.644820    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-g5mh6"]
Sep 18 13:15:39.645006 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.644980    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hp46h"
Sep 18 13:15:39.648602 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.648368    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 13:15:39.648726 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.648612    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 13:15:39.648968 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.648953    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5qlvl\""
Sep 18 13:15:39.665038 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.665016    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hp46h"]
Sep 18 13:15:39.665130 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.665049    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g5mh6"]
Sep 18 13:15:39.665169 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.665132    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:15:39.671031 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.671009    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 13:15:39.671158 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.671041    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 13:15:39.671158 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.671054    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 13:15:39.671158 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.671086    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 13:15:39.671317 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.671044    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p2lqc\""
Sep 18 13:15:39.672949 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.672930    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-crvdj"]
Sep 18 13:15:39.695990 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.695966    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-crvdj"
Sep 18 13:15:39.700698 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.700674    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 13:15:39.700821 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.700718    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 13:15:39.700821 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.700778    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 13:15:39.701040 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.701027    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8m5tk\""
Sep 18 13:15:39.701330 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.701312    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-crvdj"]
Sep 18 13:15:39.814278 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.814173    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/617fc06c-1fc8-479a-badc-dfa07662e6b0-tmp-dir\") pod \"dns-default-hp46h\" (UID: \"617fc06c-1fc8-479a-badc-dfa07662e6b0\") " pod="openshift-dns/dns-default-hp46h"
Sep 18 13:15:39.814278 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.814240    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smddj\" (UniqueName: \"kubernetes.io/projected/d0e99754-f349-4a91-8b6e-5f9622418657-kube-api-access-smddj\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:15:39.814278 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.814272    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk86f\" (UniqueName: \"kubernetes.io/projected/186f3f17-3593-4843-985c-b4bb77d3f28c-kube-api-access-rk86f\") pod \"ingress-canary-crvdj\" (UID: \"186f3f17-3593-4843-985c-b4bb77d3f28c\") " pod="openshift-ingress-canary/ingress-canary-crvdj"
Sep 18 13:15:39.814517 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.814314    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d0e99754-f349-4a91-8b6e-5f9622418657-crio-socket\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:15:39.814517 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.814337    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0e99754-f349-4a91-8b6e-5f9622418657-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:15:39.814517 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.814383    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/617fc06c-1fc8-479a-badc-dfa07662e6b0-metrics-tls\") pod \"dns-default-hp46h\" (UID: \"617fc06c-1fc8-479a-badc-dfa07662e6b0\") " pod="openshift-dns/dns-default-hp46h"
Sep 18 13:15:39.814517 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.814406    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0e99754-f349-4a91-8b6e-5f9622418657-data-volume\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:15:39.814517 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.814437    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/617fc06c-1fc8-479a-badc-dfa07662e6b0-config-volume\") pod \"dns-default-hp46h\" (UID: \"617fc06c-1fc8-479a-badc-dfa07662e6b0\") " pod="openshift-dns/dns-default-hp46h"
Sep 18 13:15:39.814517 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.814459    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcf8b\" (UniqueName: \"kubernetes.io/projected/617fc06c-1fc8-479a-badc-dfa07662e6b0-kube-api-access-qcf8b\") pod \"dns-default-hp46h\" (UID: \"617fc06c-1fc8-479a-badc-dfa07662e6b0\") " pod="openshift-dns/dns-default-hp46h"
Sep 18 13:15:39.814517 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.814493    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0e99754-f349-4a91-8b6e-5f9622418657-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:15:39.814725 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.814544    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/186f3f17-3593-4843-985c-b4bb77d3f28c-cert\") pod \"ingress-canary-crvdj\" (UID: \"186f3f17-3593-4843-985c-b4bb77d3f28c\") " pod="openshift-ingress-canary/ingress-canary-crvdj"
Sep 18 13:15:39.914830 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.914809    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk86f\" (UniqueName: \"kubernetes.io/projected/186f3f17-3593-4843-985c-b4bb77d3f28c-kube-api-access-rk86f\") pod \"ingress-canary-crvdj\" (UID: \"186f3f17-3593-4843-985c-b4bb77d3f28c\") " pod="openshift-ingress-canary/ingress-canary-crvdj"
Sep 18 13:15:39.914956 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.914844    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d0e99754-f349-4a91-8b6e-5f9622418657-crio-socket\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:15:39.914956 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.914853    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s99br_7721f73b-5184-4404-b65b-774dfff4df76/node-ca/0.log"
Sep 18 13:15:39.914956 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.914862    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0e99754-f349-4a91-8b6e-5f9622418657-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:15:39.914956 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.914942    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/617fc06c-1fc8-479a-badc-dfa07662e6b0-metrics-tls\") pod \"dns-default-hp46h\" (UID: \"617fc06c-1fc8-479a-badc-dfa07662e6b0\") " pod="openshift-dns/dns-default-hp46h"
Sep 18 13:15:39.915193 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.914968    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0e99754-f349-4a91-8b6e-5f9622418657-data-volume\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:15:39.915193 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:39.914996    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:15:39.915193 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.915009    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/617fc06c-1fc8-479a-badc-dfa07662e6b0-config-volume\") pod \"dns-default-hp46h\" (UID: \"617fc06c-1fc8-479a-badc-dfa07662e6b0\") " pod="openshift-dns/dns-default-hp46h"
Sep 18 13:15:39.915193 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.915029    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcf8b\" (UniqueName: \"kubernetes.io/projected/617fc06c-1fc8-479a-badc-dfa07662e6b0-kube-api-access-qcf8b\") pod \"dns-default-hp46h\" (UID: \"617fc06c-1fc8-479a-badc-dfa07662e6b0\") " pod="openshift-dns/dns-default-hp46h"
Sep 18 13:15:39.915193 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:39.915049    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:15:39.915193 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:39.915061    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d0e99754-f349-4a91-8b6e-5f9622418657-kube-rbac-proxy-cm podName:d0e99754-f349-4a91-8b6e-5f9622418657 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:40.415041368 +0000 UTC m=+37.770501174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d0e99754-f349-4a91-8b6e-5f9622418657-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g5mh6" (UID: "d0e99754-f349-4a91-8b6e-5f9622418657") : configmap "kube-rbac-proxy" not found
Sep 18 13:15:39.915193 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.915080    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d0e99754-f349-4a91-8b6e-5f9622418657-crio-socket\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:15:39.915193 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.915083    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0e99754-f349-4a91-8b6e-5f9622418657-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:15:39.915193 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:39.915087    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/617fc06c-1fc8-479a-badc-dfa07662e6b0-metrics-tls podName:617fc06c-1fc8-479a-badc-dfa07662e6b0 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:40.415076611 +0000 UTC m=+37.770536438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/617fc06c-1fc8-479a-badc-dfa07662e6b0-metrics-tls") pod "dns-default-hp46h" (UID: "617fc06c-1fc8-479a-badc-dfa07662e6b0") : secret "dns-default-metrics-tls" not found
Sep 18 13:15:39.915193 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:39.915148    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:15:39.915193 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:39.915181    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0e99754-f349-4a91-8b6e-5f9622418657-insights-runtime-extractor-tls podName:d0e99754-f349-4a91-8b6e-5f9622418657 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:40.415170246 +0000 UTC m=+37.770630049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0e99754-f349-4a91-8b6e-5f9622418657-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g5mh6" (UID: "d0e99754-f349-4a91-8b6e-5f9622418657") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:15:39.915193 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.915140    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/186f3f17-3593-4843-985c-b4bb77d3f28c-cert\") pod \"ingress-canary-crvdj\" (UID: \"186f3f17-3593-4843-985c-b4bb77d3f28c\") " pod="openshift-ingress-canary/ingress-canary-crvdj"
Sep 18 13:15:39.915770 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.915239    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/617fc06c-1fc8-479a-badc-dfa07662e6b0-tmp-dir\") pod \"dns-default-hp46h\" (UID: \"617fc06c-1fc8-479a-badc-dfa07662e6b0\") " pod="openshift-dns/dns-default-hp46h"
Sep 18 13:15:39.915770 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.915267    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smddj\" (UniqueName: \"kubernetes.io/projected/d0e99754-f349-4a91-8b6e-5f9622418657-kube-api-access-smddj\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:15:39.915770 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.915440    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0e99754-f349-4a91-8b6e-5f9622418657-data-volume\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:15:39.915770 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.915617    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/617fc06c-1fc8-479a-badc-dfa07662e6b0-tmp-dir\") pod \"dns-default-hp46h\" (UID: \"617fc06c-1fc8-479a-badc-dfa07662e6b0\") " pod="openshift-dns/dns-default-hp46h"
Sep 18 13:15:39.915770 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.915669    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/617fc06c-1fc8-479a-badc-dfa07662e6b0-config-volume\") pod \"dns-default-hp46h\" (UID: \"617fc06c-1fc8-479a-badc-dfa07662e6b0\") " pod="openshift-dns/dns-default-hp46h"
Sep 18 13:15:39.921325 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.921302    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/186f3f17-3593-4843-985c-b4bb77d3f28c-cert\") pod \"ingress-canary-crvdj\" (UID: \"186f3f17-3593-4843-985c-b4bb77d3f28c\") " pod="openshift-ingress-canary/ingress-canary-crvdj"
Sep 18 13:15:39.930018 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.929996    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smddj\" (UniqueName: \"kubernetes.io/projected/d0e99754-f349-4a91-8b6e-5f9622418657-kube-api-access-smddj\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:15:39.934404 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.934382    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk86f\" (UniqueName: \"kubernetes.io/projected/186f3f17-3593-4843-985c-b4bb77d3f28c-kube-api-access-rk86f\") pod \"ingress-canary-crvdj\" (UID: \"186f3f17-3593-4843-985c-b4bb77d3f28c\") " pod="openshift-ingress-canary/ingress-canary-crvdj"
Sep 18 13:15:39.934684 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:39.934663    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcf8b\" (UniqueName: \"kubernetes.io/projected/617fc06c-1fc8-479a-badc-dfa07662e6b0-kube-api-access-qcf8b\") pod \"dns-default-hp46h\" (UID: \"617fc06c-1fc8-479a-badc-dfa07662e6b0\") " pod="openshift-dns/dns-default-hp46h"
Sep 18 13:15:40.004951 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.004927    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-crvdj"
Sep 18 13:15:40.126458 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.126423    2586 generic.go:356] "Generic (PLEG): container finished" podID="af23cea6-b0da-4b1c-a704-d24956242e46" containerID="95e9d7b21ea19b04e343110fc21bda987f81470bcb12611eea17cd408c6b3b39" exitCode=0
Sep 18 13:15:40.126589 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.126469    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kv86r" event={"ID":"af23cea6-b0da-4b1c-a704-d24956242e46","Type":"ContainerDied","Data":"95e9d7b21ea19b04e343110fc21bda987f81470bcb12611eea17cd408c6b3b39"}
Sep 18 13:15:40.287551 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.287516    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-crvdj"]
Sep 18 13:15:40.291530 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:40.291497    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186f3f17_3593_4843_985c_b4bb77d3f28c.slice/crio-b057f10069f72c70c495e3770a8a3f0a417e460f422403517379d620bd1bf1f3 WatchSource:0}: Error finding container b057f10069f72c70c495e3770a8a3f0a417e460f422403517379d620bd1bf1f3: Status 404 returned error can't find the container with id b057f10069f72c70c495e3770a8a3f0a417e460f422403517379d620bd1bf1f3
Sep 18 13:15:40.419409 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.419379    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0e99754-f349-4a91-8b6e-5f9622418657-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:15:40.419534 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.419423    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/617fc06c-1fc8-479a-badc-dfa07662e6b0-metrics-tls\") pod \"dns-default-hp46h\" (UID: \"617fc06c-1fc8-479a-badc-dfa07662e6b0\") " pod="openshift-dns/dns-default-hp46h"
Sep 18 13:15:40.419534 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.419450    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0e99754-f349-4a91-8b6e-5f9622418657-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:15:40.419534 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:40.419517    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:15:40.419624 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:40.419540    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:15:40.419624 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:40.419542    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:15:40.419624 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:40.419589    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d0e99754-f349-4a91-8b6e-5f9622418657-kube-rbac-proxy-cm podName:d0e99754-f349-4a91-8b6e-5f9622418657 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:41.419574411 +0000 UTC m=+38.775034218 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d0e99754-f349-4a91-8b6e-5f9622418657-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g5mh6" (UID: "d0e99754-f349-4a91-8b6e-5f9622418657") : configmap "kube-rbac-proxy" not found
Sep 18 13:15:40.419624 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:40.419603    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/617fc06c-1fc8-479a-badc-dfa07662e6b0-metrics-tls podName:617fc06c-1fc8-479a-badc-dfa07662e6b0 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:41.419597001 +0000 UTC m=+38.775056803 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/617fc06c-1fc8-479a-badc-dfa07662e6b0-metrics-tls") pod "dns-default-hp46h" (UID: "617fc06c-1fc8-479a-badc-dfa07662e6b0") : secret "dns-default-metrics-tls" not found
Sep 18 13:15:40.419624 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:40.419613    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0e99754-f349-4a91-8b6e-5f9622418657-insights-runtime-extractor-tls podName:d0e99754-f349-4a91-8b6e-5f9622418657 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:41.419607743 +0000 UTC m=+38.775067546 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0e99754-f349-4a91-8b6e-5f9622418657-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g5mh6" (UID: "d0e99754-f349-4a91-8b6e-5f9622418657") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:15:40.559166 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.559128    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-9q8d4"]
Sep 18 13:15:40.561211 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.561195    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9q8d4"
Sep 18 13:15:40.564400 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.564378    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-q55wn\""
Sep 18 13:15:40.565334 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.565315    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 18 13:15:40.565595 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.565581    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 18 13:15:40.618109 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.618028    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-9q8d4"]
Sep 18 13:15:40.722204 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.722167    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb89g\" (UniqueName: \"kubernetes.io/projected/ea66c1bc-2ed4-436e-ade9-3545d118cd80-kube-api-access-qb89g\") pod \"migrator-76794cbd6c-9q8d4\" (UID: \"ea66c1bc-2ed4-436e-ade9-3545d118cd80\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9q8d4"
Sep 18 13:15:40.823233 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.823051    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb89g\" (UniqueName: \"kubernetes.io/projected/ea66c1bc-2ed4-436e-ade9-3545d118cd80-kube-api-access-qb89g\") pod \"migrator-76794cbd6c-9q8d4\" (UID: \"ea66c1bc-2ed4-436e-ade9-3545d118cd80\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9q8d4"
Sep 18 13:15:40.843352 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.843327    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:15:40.843527 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.843494    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:15:40.843670 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.843641    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:15:40.845585 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.845566    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb89g\" (UniqueName: \"kubernetes.io/projected/ea66c1bc-2ed4-436e-ade9-3545d118cd80-kube-api-access-qb89g\") pod \"migrator-76794cbd6c-9q8d4\" (UID: \"ea66c1bc-2ed4-436e-ade9-3545d118cd80\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9q8d4"
Sep 18 13:15:40.846993 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.846974    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 13:15:40.846993 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.846985    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 13:15:40.847115 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.846985    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 13:15:40.851613 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.847730    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 13:15:40.851613 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.848063    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-crtpr\""
Sep 18 13:15:40.851613 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.848265    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4rmbt\""
Sep 18 13:15:40.870482 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:40.870389    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9q8d4"
Sep 18 13:15:41.130407 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:41.130330    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-crvdj" event={"ID":"186f3f17-3593-4843-985c-b4bb77d3f28c","Type":"ContainerStarted","Data":"b057f10069f72c70c495e3770a8a3f0a417e460f422403517379d620bd1bf1f3"}
Sep 18 13:15:41.134264 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:41.134233    2586 generic.go:356] "Generic (PLEG): container finished" podID="af23cea6-b0da-4b1c-a704-d24956242e46" containerID="424e1107c4e04128a7360d4d3d17552ea39290bc49124f0ca29a10209489ba7c" exitCode=0
Sep 18 13:15:41.134393 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:41.134270    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kv86r" event={"ID":"af23cea6-b0da-4b1c-a704-d24956242e46","Type":"ContainerDied","Data":"424e1107c4e04128a7360d4d3d17552ea39290bc49124f0ca29a10209489ba7c"}
Sep 18 13:15:41.160479 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:41.159852    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-9q8d4"]
Sep 18 13:15:41.166985 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:41.166955    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea66c1bc_2ed4_436e_ade9_3545d118cd80.slice/crio-e3463f55d7c4873c3092b5995667a6d2623f6866ebad237f94fc25ce81b2e07b WatchSource:0}: Error finding container e3463f55d7c4873c3092b5995667a6d2623f6866ebad237f94fc25ce81b2e07b: Status 404 returned error can't find the container with id e3463f55d7c4873c3092b5995667a6d2623f6866ebad237f94fc25ce81b2e07b
Sep 18 13:15:41.428856 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:41.428816    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0e99754-f349-4a91-8b6e-5f9622418657-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:15:41.429048 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:41.428885    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/617fc06c-1fc8-479a-badc-dfa07662e6b0-metrics-tls\") pod \"dns-default-hp46h\" (UID: \"617fc06c-1fc8-479a-badc-dfa07662e6b0\") " pod="openshift-dns/dns-default-hp46h"
Sep 18 13:15:41.429048 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:41.428917    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:15:41.429048 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:41.428939    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0e99754-f349-4a91-8b6e-5f9622418657-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:15:41.429048 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:41.428994    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d0e99754-f349-4a91-8b6e-5f9622418657-kube-rbac-proxy-cm podName:d0e99754-f349-4a91-8b6e-5f9622418657 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:43.428968927 +0000 UTC m=+40.784428730 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d0e99754-f349-4a91-8b6e-5f9622418657-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g5mh6" (UID: "d0e99754-f349-4a91-8b6e-5f9622418657") : configmap "kube-rbac-proxy" not found
Sep 18 13:15:41.429277 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:41.429054    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:15:41.429277 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:41.429079    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:15:41.429277 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:41.429147    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/617fc06c-1fc8-479a-badc-dfa07662e6b0-metrics-tls podName:617fc06c-1fc8-479a-badc-dfa07662e6b0 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:43.429121189 +0000 UTC m=+40.784581007 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/617fc06c-1fc8-479a-badc-dfa07662e6b0-metrics-tls") pod "dns-default-hp46h" (UID: "617fc06c-1fc8-479a-badc-dfa07662e6b0") : secret "dns-default-metrics-tls" not found
Sep 18 13:15:41.429277 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:41.429191    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0e99754-f349-4a91-8b6e-5f9622418657-insights-runtime-extractor-tls podName:d0e99754-f349-4a91-8b6e-5f9622418657 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:43.429180202 +0000 UTC m=+40.784640005 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0e99754-f349-4a91-8b6e-5f9622418657-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g5mh6" (UID: "d0e99754-f349-4a91-8b6e-5f9622418657") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:15:42.138296 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:42.138259    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9q8d4" event={"ID":"ea66c1bc-2ed4-436e-ade9-3545d118cd80","Type":"ContainerStarted","Data":"e3463f55d7c4873c3092b5995667a6d2623f6866ebad237f94fc25ce81b2e07b"}
Sep 18 13:15:42.143155 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:42.143124    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kv86r" event={"ID":"af23cea6-b0da-4b1c-a704-d24956242e46","Type":"ContainerStarted","Data":"5ada978b96d74a55e654b59043194156ae01a42b4a14f647b8c038b5f95326c5"}
Sep 18 13:15:42.169161 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:42.169109    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kv86r" podStartSLOduration=7.46925765 podStartE2EDuration="37.169093719s" totalImagesPullingTime="29.699836069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:15:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:42.167879419 +0000 UTC m=+39.523339259" watchObservedRunningTime="2026-09-18 13:15:42.169093719 +0000 UTC m=+39.524553543"
Sep 18 13:15:43.111059 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.111025    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-59h2d"]
Sep 18 13:15:43.113261 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.113239    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-59h2d"
Sep 18 13:15:43.117992 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.117967    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 18 13:15:43.117992 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.117991    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-snpml\""
Sep 18 13:15:43.118151 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.118038    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 18 13:15:43.118296 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.118273    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 18 13:15:43.118296 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.118273    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 18 13:15:43.127195 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.127174    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-59h2d"]
Sep 18 13:15:43.149548 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.149272    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9q8d4" event={"ID":"ea66c1bc-2ed4-436e-ade9-3545d118cd80","Type":"ContainerStarted","Data":"da9d98bc2d7ec67c66a4352f767cfbba3bd5b5efefcd8fa498c051776f47cc51"}
Sep 18 13:15:43.149548 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.149379    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9q8d4" event={"ID":"ea66c1bc-2ed4-436e-ade9-3545d118cd80","Type":"ContainerStarted","Data":"eabf77eb82795e190f59b30ed9f38db7be9fa1c57cba4974e71c89d56de54a10"}
Sep 18 13:15:43.151165 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.151141    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-crvdj" event={"ID":"186f3f17-3593-4843-985c-b4bb77d3f28c","Type":"ContainerStarted","Data":"fa8322c498ad7999a95e73b09abba16de039c9b2f2e5a7804a29ba21676abf98"}
Sep 18 13:15:43.210718 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.210632    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-crvdj" podStartSLOduration=1.584663701 podStartE2EDuration="4.210616627s" totalImagesPullingTime="2.625952926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:43.209566923 +0000 UTC m=+40.565026746" watchObservedRunningTime="2026-09-18 13:15:43.210616627 +0000 UTC m=+40.566076452"
Sep 18 13:15:43.244750 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.244720    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m7tf\" (UniqueName: \"kubernetes.io/projected/0db90e09-68db-4da8-92b2-b1d908a849e0-kube-api-access-9m7tf\") pod \"service-ca-55f6fc7b86-59h2d\" (UID: \"0db90e09-68db-4da8-92b2-b1d908a849e0\") " pod="openshift-service-ca/service-ca-55f6fc7b86-59h2d"
Sep 18 13:15:43.244933 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.244769    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0db90e09-68db-4da8-92b2-b1d908a849e0-signing-key\") pod \"service-ca-55f6fc7b86-59h2d\" (UID: \"0db90e09-68db-4da8-92b2-b1d908a849e0\") " pod="openshift-service-ca/service-ca-55f6fc7b86-59h2d"
Sep 18 13:15:43.244933 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.244892    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0db90e09-68db-4da8-92b2-b1d908a849e0-signing-cabundle\") pod \"service-ca-55f6fc7b86-59h2d\" (UID: \"0db90e09-68db-4da8-92b2-b1d908a849e0\") " pod="openshift-service-ca/service-ca-55f6fc7b86-59h2d"
Sep 18 13:15:43.346239 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.346185    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m7tf\" (UniqueName: \"kubernetes.io/projected/0db90e09-68db-4da8-92b2-b1d908a849e0-kube-api-access-9m7tf\") pod \"service-ca-55f6fc7b86-59h2d\" (UID: \"0db90e09-68db-4da8-92b2-b1d908a849e0\") " pod="openshift-service-ca/service-ca-55f6fc7b86-59h2d"
Sep 18 13:15:43.346239 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.346238    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0db90e09-68db-4da8-92b2-b1d908a849e0-signing-key\") pod \"service-ca-55f6fc7b86-59h2d\" (UID: \"0db90e09-68db-4da8-92b2-b1d908a849e0\") " pod="openshift-service-ca/service-ca-55f6fc7b86-59h2d"
Sep 18 13:15:43.346458 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.346430    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0db90e09-68db-4da8-92b2-b1d908a849e0-signing-cabundle\") pod \"service-ca-55f6fc7b86-59h2d\" (UID: \"0db90e09-68db-4da8-92b2-b1d908a849e0\") " pod="openshift-service-ca/service-ca-55f6fc7b86-59h2d"
Sep 18 13:15:43.347081 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.347062    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0db90e09-68db-4da8-92b2-b1d908a849e0-signing-cabundle\") pod \"service-ca-55f6fc7b86-59h2d\" (UID: \"0db90e09-68db-4da8-92b2-b1d908a849e0\") " pod="openshift-service-ca/service-ca-55f6fc7b86-59h2d"
Sep 18 13:15:43.349790 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.349771    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0db90e09-68db-4da8-92b2-b1d908a849e0-signing-key\") pod \"service-ca-55f6fc7b86-59h2d\" (UID: \"0db90e09-68db-4da8-92b2-b1d908a849e0\") " pod="openshift-service-ca/service-ca-55f6fc7b86-59h2d"
Sep 18 13:15:43.366030 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.366003    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m7tf\" (UniqueName: \"kubernetes.io/projected/0db90e09-68db-4da8-92b2-b1d908a849e0-kube-api-access-9m7tf\") pod \"service-ca-55f6fc7b86-59h2d\" (UID: \"0db90e09-68db-4da8-92b2-b1d908a849e0\") " pod="openshift-service-ca/service-ca-55f6fc7b86-59h2d"
Sep 18 13:15:43.423148 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.423113    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-59h2d"
Sep 18 13:15:43.447365 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.447340    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0e99754-f349-4a91-8b6e-5f9622418657-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:15:43.447498 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.447385    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/617fc06c-1fc8-479a-badc-dfa07662e6b0-metrics-tls\") pod \"dns-default-hp46h\" (UID: \"617fc06c-1fc8-479a-badc-dfa07662e6b0\") " pod="openshift-dns/dns-default-hp46h"
Sep 18 13:15:43.447498 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:43.447467    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:15:43.447578 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:43.447563    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:15:43.447578 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:43.447576    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d0e99754-f349-4a91-8b6e-5f9622418657-kube-rbac-proxy-cm podName:d0e99754-f349-4a91-8b6e-5f9622418657 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:47.447560691 +0000 UTC m=+44.803020498 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d0e99754-f349-4a91-8b6e-5f9622418657-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g5mh6" (UID: "d0e99754-f349-4a91-8b6e-5f9622418657") : configmap "kube-rbac-proxy" not found
Sep 18 13:15:43.447667 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:43.447611    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/617fc06c-1fc8-479a-badc-dfa07662e6b0-metrics-tls podName:617fc06c-1fc8-479a-badc-dfa07662e6b0 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:47.447592262 +0000 UTC m=+44.803052069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/617fc06c-1fc8-479a-badc-dfa07662e6b0-metrics-tls") pod "dns-default-hp46h" (UID: "617fc06c-1fc8-479a-badc-dfa07662e6b0") : secret "dns-default-metrics-tls" not found
Sep 18 13:15:43.447667 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.447645    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0e99754-f349-4a91-8b6e-5f9622418657-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:15:43.447780 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:43.447763    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:15:43.447831 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:43.447821    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0e99754-f349-4a91-8b6e-5f9622418657-insights-runtime-extractor-tls podName:d0e99754-f349-4a91-8b6e-5f9622418657 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:47.447807509 +0000 UTC m=+44.803267317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0e99754-f349-4a91-8b6e-5f9622418657-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g5mh6" (UID: "d0e99754-f349-4a91-8b6e-5f9622418657") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:15:43.702212 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.702138    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-9q8d4" podStartSLOduration=1.954778164 podStartE2EDuration="3.702120147s" totalImagesPullingTime="1.747341983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:43.243517795 +0000 UTC m=+40.598977621" watchObservedRunningTime="2026-09-18 13:15:43.702120147 +0000 UTC m=+41.057579971"
Sep 18 13:15:43.702546 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:43.702525    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-59h2d"]
Sep 18 13:15:43.703986 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:43.703959    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db90e09_68db_4da8_92b2_b1d908a849e0.slice/crio-903de1986e74435e702c4fed31e140edf5bab0a468796484e0a035df9ba98e69 WatchSource:0}: Error finding container 903de1986e74435e702c4fed31e140edf5bab0a468796484e0a035df9ba98e69: Status 404 returned error can't find the container with id 903de1986e74435e702c4fed31e140edf5bab0a468796484e0a035df9ba98e69
Sep 18 13:15:44.155229 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:44.155195    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-59h2d" event={"ID":"0db90e09-68db-4da8-92b2-b1d908a849e0","Type":"ContainerStarted","Data":"903de1986e74435e702c4fed31e140edf5bab0a468796484e0a035df9ba98e69"}
Sep 18 13:15:46.162433 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:46.162396    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-59h2d" event={"ID":"0db90e09-68db-4da8-92b2-b1d908a849e0","Type":"ContainerStarted","Data":"b1c30ee3d575ad0dd82d3e11ee7feeeaa6fe6815fdc1f9c646672ef4b1454b19"}
Sep 18 13:15:46.191713 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:46.191655    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-59h2d" podStartSLOduration=1.382086556 podStartE2EDuration="3.19163968s" totalImagesPullingTime="1.809553124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:15:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:46.189573332 +0000 UTC m=+43.545033156" watchObservedRunningTime="2026-09-18 13:15:46.19163968 +0000 UTC m=+43.547099504"
Sep 18 13:15:47.482119 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:47.482080    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0e99754-f349-4a91-8b6e-5f9622418657-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:15:47.482467 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:47.482130    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/617fc06c-1fc8-479a-badc-dfa07662e6b0-metrics-tls\") pod \"dns-default-hp46h\" (UID: \"617fc06c-1fc8-479a-badc-dfa07662e6b0\") " pod="openshift-dns/dns-default-hp46h"
Sep 18 13:15:47.482467 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:47.482214    2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 13:15:47.482467 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:47.482215    2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 13:15:47.482467 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:47.482267    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/617fc06c-1fc8-479a-badc-dfa07662e6b0-metrics-tls podName:617fc06c-1fc8-479a-badc-dfa07662e6b0 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:55.482253185 +0000 UTC m=+52.837712987 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/617fc06c-1fc8-479a-badc-dfa07662e6b0-metrics-tls") pod "dns-default-hp46h" (UID: "617fc06c-1fc8-479a-badc-dfa07662e6b0") : secret "dns-default-metrics-tls" not found
Sep 18 13:15:47.482467 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:47.482265    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0e99754-f349-4a91-8b6e-5f9622418657-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:15:47.482467 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:47.482280    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d0e99754-f349-4a91-8b6e-5f9622418657-kube-rbac-proxy-cm podName:d0e99754-f349-4a91-8b6e-5f9622418657 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:55.482273692 +0000 UTC m=+52.837733494 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d0e99754-f349-4a91-8b6e-5f9622418657-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g5mh6" (UID: "d0e99754-f349-4a91-8b6e-5f9622418657") : configmap "kube-rbac-proxy" not found
Sep 18 13:15:47.482467 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:47.482339    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:15:47.482467 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:47.482394    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0e99754-f349-4a91-8b6e-5f9622418657-insights-runtime-extractor-tls podName:d0e99754-f349-4a91-8b6e-5f9622418657 nodeName:}" failed. No retries permitted until 2026-09-18 13:15:55.482379129 +0000 UTC m=+52.837838955 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0e99754-f349-4a91-8b6e-5f9622418657-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g5mh6" (UID: "d0e99754-f349-4a91-8b6e-5f9622418657") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:15:55.542359 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:55.542313    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/617fc06c-1fc8-479a-badc-dfa07662e6b0-metrics-tls\") pod \"dns-default-hp46h\" (UID: \"617fc06c-1fc8-479a-badc-dfa07662e6b0\") " pod="openshift-dns/dns-default-hp46h"
Sep 18 13:15:55.542974 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:55.542380    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0e99754-f349-4a91-8b6e-5f9622418657-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:15:55.542974 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:55.542476    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0e99754-f349-4a91-8b6e-5f9622418657-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:15:55.542974 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:55.542513    2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 13:15:55.542974 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:15:55.542585    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0e99754-f349-4a91-8b6e-5f9622418657-insights-runtime-extractor-tls podName:d0e99754-f349-4a91-8b6e-5f9622418657 nodeName:}" failed. No retries permitted until 2026-09-18 13:16:11.542563432 +0000 UTC m=+68.898023249 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d0e99754-f349-4a91-8b6e-5f9622418657-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g5mh6" (UID: "d0e99754-f349-4a91-8b6e-5f9622418657") : secret "insights-runtime-extractor-tls" not found
Sep 18 13:15:55.543196 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:55.543104    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d0e99754-f349-4a91-8b6e-5f9622418657-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:15:55.544815 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:55.544796    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/617fc06c-1fc8-479a-badc-dfa07662e6b0-metrics-tls\") pod \"dns-default-hp46h\" (UID: \"617fc06c-1fc8-479a-badc-dfa07662e6b0\") " pod="openshift-dns/dns-default-hp46h"
Sep 18 13:15:55.554784 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:55.554757    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hp46h"
Sep 18 13:15:55.891957 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:55.891927    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hp46h"]
Sep 18 13:15:55.895757 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:15:55.895732    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617fc06c_1fc8_479a_badc_dfa07662e6b0.slice/crio-31c71bcf10f03462d25cd8840ef58ac4351ac28b200f27bf98bdcfdd8574216f WatchSource:0}: Error finding container 31c71bcf10f03462d25cd8840ef58ac4351ac28b200f27bf98bdcfdd8574216f: Status 404 returned error can't find the container with id 31c71bcf10f03462d25cd8840ef58ac4351ac28b200f27bf98bdcfdd8574216f
Sep 18 13:15:56.189119 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:56.189028    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hp46h" event={"ID":"617fc06c-1fc8-479a-badc-dfa07662e6b0","Type":"ContainerStarted","Data":"31c71bcf10f03462d25cd8840ef58ac4351ac28b200f27bf98bdcfdd8574216f"}
Sep 18 13:15:59.200171 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:59.200134    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hp46h" event={"ID":"617fc06c-1fc8-479a-badc-dfa07662e6b0","Type":"ContainerStarted","Data":"f160b0a2bfa6609ac31501c73612d497a5141ca94a5fad08222c20ce7faf1a69"}
Sep 18 13:15:59.200609 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:59.200185    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hp46h" event={"ID":"617fc06c-1fc8-479a-badc-dfa07662e6b0","Type":"ContainerStarted","Data":"00f3820cdd04903654f7ebdf33f908d00210f39757d3cb8760794dba4403dfb8"}
Sep 18 13:15:59.200609 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:59.200266    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hp46h"
Sep 18 13:15:59.228940 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:15:59.228835    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hp46h" podStartSLOduration=17.970748849 podStartE2EDuration="20.228815721s" totalImagesPullingTime="2.258066872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:15:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:15:59.227302917 +0000 UTC m=+56.582762742" watchObservedRunningTime="2026-09-18 13:15:59.228815721 +0000 UTC m=+56.584275548"
Sep 18 13:16:04.116128 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:04.116098    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lcqcw"
Sep 18 13:16:09.206119 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:09.206084    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hp46h"
Sep 18 13:16:10.646264 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.646224    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e310da4-ab19-4631-9717-86fb07925256-original-pull-secret\") pod \"global-pull-secret-syncer-pc5zc\" (UID: \"7e310da4-ab19-4631-9717-86fb07925256\") " pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:16:10.646646 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.646302    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47797f8b-a736-4caa-9b4a-6de71b38c001-metrics-certs\") pod \"network-metrics-daemon-qlzxp\" (UID: \"47797f8b-a736-4caa-9b4a-6de71b38c001\") " pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:16:10.649250 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.649218    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 13:16:10.649558 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.649532    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 13:16:10.658959 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.658936    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7e310da4-ab19-4631-9717-86fb07925256-original-pull-secret\") pod \"global-pull-secret-syncer-pc5zc\" (UID: \"7e310da4-ab19-4631-9717-86fb07925256\") " pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:16:10.659054 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.658936    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47797f8b-a736-4caa-9b4a-6de71b38c001-metrics-certs\") pod \"network-metrics-daemon-qlzxp\" (UID: \"47797f8b-a736-4caa-9b4a-6de71b38c001\") " pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:16:10.747679 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.747633    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88wcc\" (UniqueName: \"kubernetes.io/projected/1b7871d5-8610-4b4a-b099-4163ff75df5e-kube-api-access-88wcc\") pod \"network-check-target-tmdn8\" (UID: \"1b7871d5-8610-4b4a-b099-4163ff75df5e\") " pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:16:10.750666 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.750648    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 13:16:10.759719 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.759703    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 13:16:10.771544 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.771519    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88wcc\" (UniqueName: \"kubernetes.io/projected/1b7871d5-8610-4b4a-b099-4163ff75df5e-kube-api-access-88wcc\") pod \"network-check-target-tmdn8\" (UID: \"1b7871d5-8610-4b4a-b099-4163ff75df5e\") " pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:16:10.857759 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.857728    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4rmbt\""
Sep 18 13:16:10.862278 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.862263    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pc5zc"
Sep 18 13:16:10.866588 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.866571    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qlzxp"
Sep 18 13:16:10.868583 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.868568    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-crtpr\""
Sep 18 13:16:10.877411 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.877367    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:16:10.902436 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.902094    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb"]
Sep 18 13:16:10.905924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.905759    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb"
Sep 18 13:16:10.912161 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.912133    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 18 13:16:10.913115 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.912385    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 13:16:10.913115 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.912578    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 18 13:16:10.913115 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.912764    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 13:16:10.913115 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.912956    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 13:16:10.913115 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.913078    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 18 13:16:10.913115 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.913078    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 18 13:16:10.931346 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.931278    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb"]
Sep 18 13:16:10.949290 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.949215    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0103a63f-4f96-48cd-b609-7e803f3bdc5c-ca\") pod \"cluster-proxy-proxy-agent-75c8cf94c-k2zdb\" (UID: \"0103a63f-4f96-48cd-b609-7e803f3bdc5c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb"
Sep 18 13:16:10.949290 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.949260    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0103a63f-4f96-48cd-b609-7e803f3bdc5c-hub\") pod \"cluster-proxy-proxy-agent-75c8cf94c-k2zdb\" (UID: \"0103a63f-4f96-48cd-b609-7e803f3bdc5c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb"
Sep 18 13:16:10.949451 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.949294    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0103a63f-4f96-48cd-b609-7e803f3bdc5c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75c8cf94c-k2zdb\" (UID: \"0103a63f-4f96-48cd-b609-7e803f3bdc5c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb"
Sep 18 13:16:10.949628 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.949471    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbbpt\" (UniqueName: \"kubernetes.io/projected/0103a63f-4f96-48cd-b609-7e803f3bdc5c-kube-api-access-pbbpt\") pod \"cluster-proxy-proxy-agent-75c8cf94c-k2zdb\" (UID: \"0103a63f-4f96-48cd-b609-7e803f3bdc5c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb"
Sep 18 13:16:10.949628 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.949531    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0103a63f-4f96-48cd-b609-7e803f3bdc5c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75c8cf94c-k2zdb\" (UID: \"0103a63f-4f96-48cd-b609-7e803f3bdc5c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb"
Sep 18 13:16:10.949628 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:10.949560    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0103a63f-4f96-48cd-b609-7e803f3bdc5c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75c8cf94c-k2zdb\" (UID: \"0103a63f-4f96-48cd-b609-7e803f3bdc5c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb"
Sep 18 13:16:11.050769 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.050732    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbbpt\" (UniqueName: \"kubernetes.io/projected/0103a63f-4f96-48cd-b609-7e803f3bdc5c-kube-api-access-pbbpt\") pod \"cluster-proxy-proxy-agent-75c8cf94c-k2zdb\" (UID: \"0103a63f-4f96-48cd-b609-7e803f3bdc5c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb"
Sep 18 13:16:11.050769 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.050772    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0103a63f-4f96-48cd-b609-7e803f3bdc5c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75c8cf94c-k2zdb\" (UID: \"0103a63f-4f96-48cd-b609-7e803f3bdc5c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb"
Sep 18 13:16:11.051094 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.050790    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0103a63f-4f96-48cd-b609-7e803f3bdc5c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75c8cf94c-k2zdb\" (UID: \"0103a63f-4f96-48cd-b609-7e803f3bdc5c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb"
Sep 18 13:16:11.051094 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.050813    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0103a63f-4f96-48cd-b609-7e803f3bdc5c-ca\") pod \"cluster-proxy-proxy-agent-75c8cf94c-k2zdb\" (UID: \"0103a63f-4f96-48cd-b609-7e803f3bdc5c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb"
Sep 18 13:16:11.051094 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.050883    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0103a63f-4f96-48cd-b609-7e803f3bdc5c-hub\") pod \"cluster-proxy-proxy-agent-75c8cf94c-k2zdb\" (UID: \"0103a63f-4f96-48cd-b609-7e803f3bdc5c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb"
Sep 18 13:16:11.051094 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.050944    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0103a63f-4f96-48cd-b609-7e803f3bdc5c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75c8cf94c-k2zdb\" (UID: \"0103a63f-4f96-48cd-b609-7e803f3bdc5c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb"
Sep 18 13:16:11.051526 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.051499    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0103a63f-4f96-48cd-b609-7e803f3bdc5c-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75c8cf94c-k2zdb\" (UID: \"0103a63f-4f96-48cd-b609-7e803f3bdc5c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb"
Sep 18 13:16:11.053441 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.053408    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0103a63f-4f96-48cd-b609-7e803f3bdc5c-ca\") pod \"cluster-proxy-proxy-agent-75c8cf94c-k2zdb\" (UID: \"0103a63f-4f96-48cd-b609-7e803f3bdc5c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb"
Sep 18 13:16:11.053554 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.053505    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0103a63f-4f96-48cd-b609-7e803f3bdc5c-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75c8cf94c-k2zdb\" (UID: \"0103a63f-4f96-48cd-b609-7e803f3bdc5c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb"
Sep 18 13:16:11.053613 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.053568    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0103a63f-4f96-48cd-b609-7e803f3bdc5c-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75c8cf94c-k2zdb\" (UID: \"0103a63f-4f96-48cd-b609-7e803f3bdc5c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb"
Sep 18 13:16:11.053613 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.053567    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0103a63f-4f96-48cd-b609-7e803f3bdc5c-hub\") pod \"cluster-proxy-proxy-agent-75c8cf94c-k2zdb\" (UID: \"0103a63f-4f96-48cd-b609-7e803f3bdc5c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb"
Sep 18 13:16:11.085418 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.085388    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbbpt\" (UniqueName: \"kubernetes.io/projected/0103a63f-4f96-48cd-b609-7e803f3bdc5c-kube-api-access-pbbpt\") pod \"cluster-proxy-proxy-agent-75c8cf94c-k2zdb\" (UID: \"0103a63f-4f96-48cd-b609-7e803f3bdc5c\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb"
Sep 18 13:16:11.140649 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.140618    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6645f89c6b-svklk"]
Sep 18 13:16:11.143171 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.143147    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.146461 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.146440    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 13:16:11.146589 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.146487    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 13:16:11.146764 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.146747    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 13:16:11.146821 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.146786    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-w5nbx\""
Sep 18 13:16:11.147574 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.147545    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pc5zc"]
Sep 18 13:16:11.154432 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.154410    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 13:16:11.156151 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:16:11.156130    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e310da4_ab19_4631_9717_86fb07925256.slice/crio-2a7c11b355a254033cf28bbff4763f8e0c4242c085ab2caf52bf54f90b5aeda7 WatchSource:0}: Error finding container 2a7c11b355a254033cf28bbff4763f8e0c4242c085ab2caf52bf54f90b5aeda7: Status 404 returned error can't find the container with id 2a7c11b355a254033cf28bbff4763f8e0c4242c085ab2caf52bf54f90b5aeda7
Sep 18 13:16:11.162239 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.162214    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6645f89c6b-svklk"]
Sep 18 13:16:11.174177 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.174155    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qlzxp"]
Sep 18 13:16:11.177759 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:16:11.177735    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47797f8b_a736_4caa_9b4a_6de71b38c001.slice/crio-dc8df58aee22fb4b5d695a99fab4c446fb600394e6b0be79668bab1e4195c954 WatchSource:0}: Error finding container dc8df58aee22fb4b5d695a99fab4c446fb600394e6b0be79668bab1e4195c954: Status 404 returned error can't find the container with id dc8df58aee22fb4b5d695a99fab4c446fb600394e6b0be79668bab1e4195c954
Sep 18 13:16:11.183750 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.183728    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tmdn8"]
Sep 18 13:16:11.186626 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:16:11.186600    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7871d5_8610_4b4a_b099_4163ff75df5e.slice/crio-5cbd5aff01a97c1448d1715f5fe45db630b33aa147507b5a53ec952a85bf4be3 WatchSource:0}: Error finding container 5cbd5aff01a97c1448d1715f5fe45db630b33aa147507b5a53ec952a85bf4be3: Status 404 returned error can't find the container with id 5cbd5aff01a97c1448d1715f5fe45db630b33aa147507b5a53ec952a85bf4be3
Sep 18 13:16:11.227192 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.227161    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb"
Sep 18 13:16:11.240363 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.240333    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tmdn8" event={"ID":"1b7871d5-8610-4b4a-b099-4163ff75df5e","Type":"ContainerStarted","Data":"5cbd5aff01a97c1448d1715f5fe45db630b33aa147507b5a53ec952a85bf4be3"}
Sep 18 13:16:11.241657 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.241629    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qlzxp" event={"ID":"47797f8b-a736-4caa-9b4a-6de71b38c001","Type":"ContainerStarted","Data":"dc8df58aee22fb4b5d695a99fab4c446fb600394e6b0be79668bab1e4195c954"}
Sep 18 13:16:11.242715 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.242695    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pc5zc" event={"ID":"7e310da4-ab19-4631-9717-86fb07925256","Type":"ContainerStarted","Data":"2a7c11b355a254033cf28bbff4763f8e0c4242c085ab2caf52bf54f90b5aeda7"}
Sep 18 13:16:11.252076 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.252052    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7fe2b0a-4e0d-483b-a53a-cd6387f695cb-trusted-ca\") pod \"image-registry-6645f89c6b-svklk\" (UID: \"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb\") " pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.252159 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.252100    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcgdx\" (UniqueName: \"kubernetes.io/projected/a7fe2b0a-4e0d-483b-a53a-cd6387f695cb-kube-api-access-wcgdx\") pod \"image-registry-6645f89c6b-svklk\" (UID: \"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb\") " pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.252159 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.252149    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7fe2b0a-4e0d-483b-a53a-cd6387f695cb-installation-pull-secrets\") pod \"image-registry-6645f89c6b-svklk\" (UID: \"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb\") " pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.252241 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.252173    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7fe2b0a-4e0d-483b-a53a-cd6387f695cb-image-registry-private-configuration\") pod \"image-registry-6645f89c6b-svklk\" (UID: \"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb\") " pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.252241 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.252208    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7fe2b0a-4e0d-483b-a53a-cd6387f695cb-bound-sa-token\") pod \"image-registry-6645f89c6b-svklk\" (UID: \"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb\") " pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.252312 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.252248    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7fe2b0a-4e0d-483b-a53a-cd6387f695cb-registry-tls\") pod \"image-registry-6645f89c6b-svklk\" (UID: \"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb\") " pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.252312 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.252281    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7fe2b0a-4e0d-483b-a53a-cd6387f695cb-ca-trust-extracted\") pod \"image-registry-6645f89c6b-svklk\" (UID: \"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb\") " pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.252312 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.252303    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7fe2b0a-4e0d-483b-a53a-cd6387f695cb-registry-certificates\") pod \"image-registry-6645f89c6b-svklk\" (UID: \"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb\") " pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.353164 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.353126    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7fe2b0a-4e0d-483b-a53a-cd6387f695cb-installation-pull-secrets\") pod \"image-registry-6645f89c6b-svklk\" (UID: \"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb\") " pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.353164 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.353166    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7fe2b0a-4e0d-483b-a53a-cd6387f695cb-image-registry-private-configuration\") pod \"image-registry-6645f89c6b-svklk\" (UID: \"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb\") " pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.353416 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.353192    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7fe2b0a-4e0d-483b-a53a-cd6387f695cb-bound-sa-token\") pod \"image-registry-6645f89c6b-svklk\" (UID: \"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb\") " pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.353416 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.353221    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7fe2b0a-4e0d-483b-a53a-cd6387f695cb-registry-tls\") pod \"image-registry-6645f89c6b-svklk\" (UID: \"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb\") " pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.353416 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.353239    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7fe2b0a-4e0d-483b-a53a-cd6387f695cb-ca-trust-extracted\") pod \"image-registry-6645f89c6b-svklk\" (UID: \"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb\") " pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.353416 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.353256    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7fe2b0a-4e0d-483b-a53a-cd6387f695cb-registry-certificates\") pod \"image-registry-6645f89c6b-svklk\" (UID: \"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb\") " pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.353613 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.353421    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7fe2b0a-4e0d-483b-a53a-cd6387f695cb-trusted-ca\") pod \"image-registry-6645f89c6b-svklk\" (UID: \"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb\") " pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.353613 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.353485    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcgdx\" (UniqueName: \"kubernetes.io/projected/a7fe2b0a-4e0d-483b-a53a-cd6387f695cb-kube-api-access-wcgdx\") pod \"image-registry-6645f89c6b-svklk\" (UID: \"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb\") " pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.353860 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.353823    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a7fe2b0a-4e0d-483b-a53a-cd6387f695cb-ca-trust-extracted\") pod \"image-registry-6645f89c6b-svklk\" (UID: \"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb\") " pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.354414 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.354388    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a7fe2b0a-4e0d-483b-a53a-cd6387f695cb-registry-certificates\") pod \"image-registry-6645f89c6b-svklk\" (UID: \"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb\") " pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.354519 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.354440    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a7fe2b0a-4e0d-483b-a53a-cd6387f695cb-trusted-ca\") pod \"image-registry-6645f89c6b-svklk\" (UID: \"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb\") " pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.355813 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.355793    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a7fe2b0a-4e0d-483b-a53a-cd6387f695cb-installation-pull-secrets\") pod \"image-registry-6645f89c6b-svklk\" (UID: \"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb\") " pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.355922 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.355870    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a7fe2b0a-4e0d-483b-a53a-cd6387f695cb-registry-tls\") pod \"image-registry-6645f89c6b-svklk\" (UID: \"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb\") " pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.355922 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.355885    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a7fe2b0a-4e0d-483b-a53a-cd6387f695cb-image-registry-private-configuration\") pod \"image-registry-6645f89c6b-svklk\" (UID: \"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb\") " pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.367180 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.367156    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a7fe2b0a-4e0d-483b-a53a-cd6387f695cb-bound-sa-token\") pod \"image-registry-6645f89c6b-svklk\" (UID: \"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb\") " pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.368028 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.368011    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcgdx\" (UniqueName: \"kubernetes.io/projected/a7fe2b0a-4e0d-483b-a53a-cd6387f695cb-kube-api-access-wcgdx\") pod \"image-registry-6645f89c6b-svklk\" (UID: \"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb\") " pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.453118 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.453035    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:11.501717 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.501353    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb"]
Sep 18 13:16:11.517477 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:16:11.517432    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0103a63f_4f96_48cd_b609_7e803f3bdc5c.slice/crio-ca009a15b0b9d478a93e83da5f1213a2741223807f13cb9ce56866782288070d WatchSource:0}: Error finding container ca009a15b0b9d478a93e83da5f1213a2741223807f13cb9ce56866782288070d: Status 404 returned error can't find the container with id ca009a15b0b9d478a93e83da5f1213a2741223807f13cb9ce56866782288070d
Sep 18 13:16:11.555194 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.555166    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0e99754-f349-4a91-8b6e-5f9622418657-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:16:11.557551 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.557524    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d0e99754-f349-4a91-8b6e-5f9622418657-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g5mh6\" (UID: \"d0e99754-f349-4a91-8b6e-5f9622418657\") " pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:16:11.739597 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.739468    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6645f89c6b-svklk"]
Sep 18 13:16:11.752261 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:16:11.752220    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7fe2b0a_4e0d_483b_a53a_cd6387f695cb.slice/crio-39ca0260a12b6028ee392ea651f343dc3584f92d395e66f84140d9cb4f5b2713 WatchSource:0}: Error finding container 39ca0260a12b6028ee392ea651f343dc3584f92d395e66f84140d9cb4f5b2713: Status 404 returned error can't find the container with id 39ca0260a12b6028ee392ea651f343dc3584f92d395e66f84140d9cb4f5b2713
Sep 18 13:16:11.778778 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.778753    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p2lqc\""
Sep 18 13:16:11.787926 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:11.787698    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g5mh6"
Sep 18 13:16:12.091293 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:12.090034    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g5mh6"]
Sep 18 13:16:12.255931 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:12.255541    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6645f89c6b-svklk" event={"ID":"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb","Type":"ContainerStarted","Data":"e2e178f26e6b24b63fe8231304c72009a49591ad612807a8643b49c256695e2c"}
Sep 18 13:16:12.255931 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:12.255634    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6645f89c6b-svklk" event={"ID":"a7fe2b0a-4e0d-483b-a53a-cd6387f695cb","Type":"ContainerStarted","Data":"39ca0260a12b6028ee392ea651f343dc3584f92d395e66f84140d9cb4f5b2713"}
Sep 18 13:16:12.256425 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:12.256404    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:12.257464 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:12.257400    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb" event={"ID":"0103a63f-4f96-48cd-b609-7e803f3bdc5c","Type":"ContainerStarted","Data":"ca009a15b0b9d478a93e83da5f1213a2741223807f13cb9ce56866782288070d"}
Sep 18 13:16:12.279133 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:12.278668    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6645f89c6b-svklk" podStartSLOduration=1.278644724 podStartE2EDuration="1.278644724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:16:12.277284451 +0000 UTC m=+69.632744268" watchObservedRunningTime="2026-09-18 13:16:12.278644724 +0000 UTC m=+69.634104551"
Sep 18 13:16:12.385345 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:16:12.385267    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e99754_f349_4a91_8b6e_5f9622418657.slice/crio-75de18c961e591e1a93db052b17d2cf3b8e213ec8aec67095035e809a358fe2a WatchSource:0}: Error finding container 75de18c961e591e1a93db052b17d2cf3b8e213ec8aec67095035e809a358fe2a: Status 404 returned error can't find the container with id 75de18c961e591e1a93db052b17d2cf3b8e213ec8aec67095035e809a358fe2a
Sep 18 13:16:13.267490 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:13.267250    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qlzxp" event={"ID":"47797f8b-a736-4caa-9b4a-6de71b38c001","Type":"ContainerStarted","Data":"943fabe6a74a45a3f2116a526ec2e2ec79ee29e9e13e0dcd2393ac20391094cc"}
Sep 18 13:16:13.267490 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:13.267316    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qlzxp" event={"ID":"47797f8b-a736-4caa-9b4a-6de71b38c001","Type":"ContainerStarted","Data":"5ae3b967955e656ab127c3266833617c1b1efb03cc48d47f4b364889915a5483"}
Sep 18 13:16:13.273822 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:13.273585    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g5mh6" event={"ID":"d0e99754-f349-4a91-8b6e-5f9622418657","Type":"ContainerStarted","Data":"54cb01aa5816be5118d2b0fddaf4e6ef2839535319a119ac6896af9a464e8705"}
Sep 18 13:16:13.273822 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:13.273642    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g5mh6" event={"ID":"d0e99754-f349-4a91-8b6e-5f9622418657","Type":"ContainerStarted","Data":"75de18c961e591e1a93db052b17d2cf3b8e213ec8aec67095035e809a358fe2a"}
Sep 18 13:16:13.286262 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:13.285537    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qlzxp" podStartSLOduration=68.046220555 podStartE2EDuration="1m9.285516286s" totalImagesPullingTime="1.239295731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:16:13.284217938 +0000 UTC m=+70.639677763" watchObservedRunningTime="2026-09-18 13:16:13.285516286 +0000 UTC m=+70.640976112"
Sep 18 13:16:14.281086 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:14.280997    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g5mh6" event={"ID":"d0e99754-f349-4a91-8b6e-5f9622418657","Type":"ContainerStarted","Data":"8ad779bbc20b5810b8175a4ba860f8c0d8ca6d0f903b77c22fdf2828e6d8e5e3"}
Sep 18 13:16:19.300552 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:19.300517    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pc5zc" event={"ID":"7e310da4-ab19-4631-9717-86fb07925256","Type":"ContainerStarted","Data":"e94824cbb2ea1f2762cd75c94c69d1947f7ccc34eae57630c8ef42351e2896e3"}
Sep 18 13:16:19.302090 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:19.302056    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tmdn8" event={"ID":"1b7871d5-8610-4b4a-b099-4163ff75df5e","Type":"ContainerStarted","Data":"cdb340e2e1305b1954a3f4abbeaad33fc2616efc7ad17b60b2a98d44d08462bc"}
Sep 18 13:16:19.302188 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:19.302108    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:16:19.304108 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:19.304087    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb" event={"ID":"0103a63f-4f96-48cd-b609-7e803f3bdc5c","Type":"ContainerStarted","Data":"885b1e53088db7e8ce476f1ce38142fc77da81d7ce341b05a075fdd99f9434d2"}
Sep 18 13:16:19.304224 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:19.304120    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb" event={"ID":"0103a63f-4f96-48cd-b609-7e803f3bdc5c","Type":"ContainerStarted","Data":"ba939313aa7946e784abe56c000d0a0537ad250a4d23ceb212a0e4edb45b7078"}
Sep 18 13:16:19.304224 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:19.304141    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb" event={"ID":"0103a63f-4f96-48cd-b609-7e803f3bdc5c","Type":"ContainerStarted","Data":"5b15f6974a373fe8b2c23cc18da484b24334aee6a4a5fe64b810ea6c3b8cbe18"}
Sep 18 13:16:19.306125 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:19.306104    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g5mh6" event={"ID":"d0e99754-f349-4a91-8b6e-5f9622418657","Type":"ContainerStarted","Data":"09a19e7ec960345d757dac99037249c294e420b8943a486ad863d5242dedc706"}
Sep 18 13:16:19.333153 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:19.333101    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pc5zc" podStartSLOduration=67.046086188 podStartE2EDuration="1m14.333083086s" totalImagesPullingTime="7.286996898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:16:19.332498593 +0000 UTC m=+76.687958418" watchObservedRunningTime="2026-09-18 13:16:19.333083086 +0000 UTC m=+76.688542912"
Sep 18 13:16:19.365016 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:19.364957    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tmdn8" podStartSLOduration=67.108322905 podStartE2EDuration="1m14.364921133s" totalImagesPullingTime="7.256598228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:16:19.359480641 +0000 UTC m=+76.714940477" watchObservedRunningTime="2026-09-18 13:16:19.364921133 +0000 UTC m=+76.720380949"
Sep 18 13:16:19.383045 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:19.383005    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-g5mh6" podStartSLOduration=34.483092175 podStartE2EDuration="40.382989054s" totalImagesPullingTime="5.899896879s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:15:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:16:19.382667343 +0000 UTC m=+76.738127167" watchObservedRunningTime="2026-09-18 13:16:19.382989054 +0000 UTC m=+76.738448879"
Sep 18 13:16:19.410747 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:19.410689    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75c8cf94c-k2zdb" podStartSLOduration=2.4863527420000002 podStartE2EDuration="9.410671488s" totalImagesPullingTime="6.924318746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:16:19.409584799 +0000 UTC m=+76.765044624" watchObservedRunningTime="2026-09-18 13:16:19.410671488 +0000 UTC m=+76.766131313"
Sep 18 13:16:23.222556 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.222521    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-b6t2b"]
Sep 18 13:16:23.224558 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.224541    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.226840 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.226820    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 13:16:23.227733 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.227669    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 13:16:23.227733 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.227679    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 13:16:23.228092 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.228064    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 13:16:23.228172 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.228115    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 13:16:23.228267 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.228251    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 13:16:23.228424 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.228403    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p6v7j\""
Sep 18 13:16:23.340250 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.340215    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-node-exporter-tls\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.340405 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.340261    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-node-exporter-wtmp\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.340405 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.340293    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-node-exporter-accelerators-collector-config\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.340405 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.340379    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.340543 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.340438    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-sys\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.340543 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.340472    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-metrics-client-ca\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.340543 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.340513    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-root\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.340543 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.340537    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbt5f\" (UniqueName: \"kubernetes.io/projected/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-kube-api-access-pbt5f\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.340675 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.340597    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-node-exporter-textfile\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.441826 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.441792    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-node-exporter-tls\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.442041 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.441834    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-node-exporter-wtmp\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.442041 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.441865    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-node-exporter-accelerators-collector-config\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.442041 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.441930    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.442041 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.441977    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-sys\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.442041 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:16:23.441984    2586 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 18 13:16:23.442041 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.441999    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-metrics-client-ca\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.442334 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.442041    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-node-exporter-wtmp\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.442334 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:16:23.442059    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-node-exporter-tls podName:9e5f1d5b-d9ca-4a0b-985b-4b0147312765 nodeName:}" failed. No retries permitted until 2026-09-18 13:16:23.942037164 +0000 UTC m=+81.297496969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-node-exporter-tls") pod "node-exporter-b6t2b" (UID: "9e5f1d5b-d9ca-4a0b-985b-4b0147312765") : secret "node-exporter-tls" not found
Sep 18 13:16:23.442334 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.442105    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-sys\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.442334 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.442169    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-root\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.442334 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.442207    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbt5f\" (UniqueName: \"kubernetes.io/projected/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-kube-api-access-pbt5f\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.442334 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.442249    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-node-exporter-textfile\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.442334 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.442324    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-root\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.442725 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.442579    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-node-exporter-textfile\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.442725 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.442604    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-node-exporter-accelerators-collector-config\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.442725 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.442627    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-metrics-client-ca\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.444894 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.444870    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.464780 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.464750    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbt5f\" (UniqueName: \"kubernetes.io/projected/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-kube-api-access-pbt5f\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.946920 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.946877    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-node-exporter-tls\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:23.949409 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:23.949371    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9e5f1d5b-d9ca-4a0b-985b-4b0147312765-node-exporter-tls\") pod \"node-exporter-b6t2b\" (UID: \"9e5f1d5b-d9ca-4a0b-985b-4b0147312765\") " pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:24.134439 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:24.134403    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b6t2b"
Sep 18 13:16:24.147362 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:16:24.147335    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5f1d5b_d9ca_4a0b_985b_4b0147312765.slice/crio-19bbe80ee464521c5f8d28f700cf45ec31fa24d488de648ae1caf78518e8d159 WatchSource:0}: Error finding container 19bbe80ee464521c5f8d28f700cf45ec31fa24d488de648ae1caf78518e8d159: Status 404 returned error can't find the container with id 19bbe80ee464521c5f8d28f700cf45ec31fa24d488de648ae1caf78518e8d159
Sep 18 13:16:24.326778 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:24.326694    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b6t2b" event={"ID":"9e5f1d5b-d9ca-4a0b-985b-4b0147312765","Type":"ContainerStarted","Data":"19bbe80ee464521c5f8d28f700cf45ec31fa24d488de648ae1caf78518e8d159"}
Sep 18 13:16:25.332408 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:25.332375    2586 generic.go:356] "Generic (PLEG): container finished" podID="9e5f1d5b-d9ca-4a0b-985b-4b0147312765" containerID="1cdfd79b469a17b691a58fff53e42f306b0d9977d3237f6ad34a103dd4ab7686" exitCode=0
Sep 18 13:16:25.332780 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:25.332422    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b6t2b" event={"ID":"9e5f1d5b-d9ca-4a0b-985b-4b0147312765","Type":"ContainerDied","Data":"1cdfd79b469a17b691a58fff53e42f306b0d9977d3237f6ad34a103dd4ab7686"}
Sep 18 13:16:26.339159 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:26.339119    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b6t2b" event={"ID":"9e5f1d5b-d9ca-4a0b-985b-4b0147312765","Type":"ContainerStarted","Data":"32aaf7e7d0fe475d0f34d10b5bc338184af81ea12b168c26e55d765c1f06b8a3"}
Sep 18 13:16:26.339538 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:26.339179    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b6t2b" event={"ID":"9e5f1d5b-d9ca-4a0b-985b-4b0147312765","Type":"ContainerStarted","Data":"0cef639782c76fa1f9d5f3dd135c07ddef27bfbc9a8b19dc3a36ea70522390ce"}
Sep 18 13:16:26.362491 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:26.362436    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-b6t2b" podStartSLOduration=2.297396934 podStartE2EDuration="3.36241418s" totalImagesPullingTime="995.647885ms" totalInitContainerRuntime="69.369361ms" isStatefulPod=true podCreationTimestamp="2026-09-18 13:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:16:26.361078814 +0000 UTC m=+83.716538639" watchObservedRunningTime="2026-09-18 13:16:26.36241418 +0000 UTC m=+83.717874005"
Sep 18 13:16:27.673758 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.673723    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-76bb975779-mgnrc"]
Sep 18 13:16:27.678149 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.678132    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:16:27.681729 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.681706    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 13:16:27.681729 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.681706    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 18 13:16:27.681729 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.681727    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 18 13:16:27.681941 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.681735    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-njxj4\""
Sep 18 13:16:27.682162 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.682130    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 18 13:16:27.682243 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.682211    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1ogd6tq5taeaa\""
Sep 18 13:16:27.704467 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.704440    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76bb975779-mgnrc"]
Sep 18 13:16:27.777029 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.776999    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff6pk\" (UniqueName: \"kubernetes.io/projected/c7cd73e0-5d9c-46be-9532-5661607b7b39-kube-api-access-ff6pk\") pod \"metrics-server-76bb975779-mgnrc\" (UID: \"c7cd73e0-5d9c-46be-9532-5661607b7b39\") " pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:16:27.777206 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.777037    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7cd73e0-5d9c-46be-9532-5661607b7b39-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76bb975779-mgnrc\" (UID: \"c7cd73e0-5d9c-46be-9532-5661607b7b39\") " pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:16:27.777206 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.777062    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c7cd73e0-5d9c-46be-9532-5661607b7b39-audit-log\") pod \"metrics-server-76bb975779-mgnrc\" (UID: \"c7cd73e0-5d9c-46be-9532-5661607b7b39\") " pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:16:27.777206 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.777103    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c7cd73e0-5d9c-46be-9532-5661607b7b39-secret-metrics-server-tls\") pod \"metrics-server-76bb975779-mgnrc\" (UID: \"c7cd73e0-5d9c-46be-9532-5661607b7b39\") " pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:16:27.777206 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.777181    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c7cd73e0-5d9c-46be-9532-5661607b7b39-client-ca-bundle\") pod \"metrics-server-76bb975779-mgnrc\" (UID: \"c7cd73e0-5d9c-46be-9532-5661607b7b39\") " pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:16:27.777382 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.777228    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c7cd73e0-5d9c-46be-9532-5661607b7b39-metrics-server-audit-profiles\") pod \"metrics-server-76bb975779-mgnrc\" (UID: \"c7cd73e0-5d9c-46be-9532-5661607b7b39\") " pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:16:27.777382 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.777257    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c7cd73e0-5d9c-46be-9532-5661607b7b39-secret-metrics-server-client-certs\") pod \"metrics-server-76bb975779-mgnrc\" (UID: \"c7cd73e0-5d9c-46be-9532-5661607b7b39\") " pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:16:27.877741 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.877710    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff6pk\" (UniqueName: \"kubernetes.io/projected/c7cd73e0-5d9c-46be-9532-5661607b7b39-kube-api-access-ff6pk\") pod \"metrics-server-76bb975779-mgnrc\" (UID: \"c7cd73e0-5d9c-46be-9532-5661607b7b39\") " pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:16:27.877934 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.877752    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7cd73e0-5d9c-46be-9532-5661607b7b39-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76bb975779-mgnrc\" (UID: \"c7cd73e0-5d9c-46be-9532-5661607b7b39\") " pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:16:27.877934 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.877780    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c7cd73e0-5d9c-46be-9532-5661607b7b39-audit-log\") pod \"metrics-server-76bb975779-mgnrc\" (UID: \"c7cd73e0-5d9c-46be-9532-5661607b7b39\") " pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:16:27.877934 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.877804    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c7cd73e0-5d9c-46be-9532-5661607b7b39-secret-metrics-server-tls\") pod \"metrics-server-76bb975779-mgnrc\" (UID: \"c7cd73e0-5d9c-46be-9532-5661607b7b39\") " pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:16:27.877934 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.877831    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c7cd73e0-5d9c-46be-9532-5661607b7b39-client-ca-bundle\") pod \"metrics-server-76bb975779-mgnrc\" (UID: \"c7cd73e0-5d9c-46be-9532-5661607b7b39\") " pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:16:27.877934 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.877852    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c7cd73e0-5d9c-46be-9532-5661607b7b39-metrics-server-audit-profiles\") pod \"metrics-server-76bb975779-mgnrc\" (UID: \"c7cd73e0-5d9c-46be-9532-5661607b7b39\") " pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:16:27.877934 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.877867    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c7cd73e0-5d9c-46be-9532-5661607b7b39-secret-metrics-server-client-certs\") pod \"metrics-server-76bb975779-mgnrc\" (UID: \"c7cd73e0-5d9c-46be-9532-5661607b7b39\") " pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:16:27.878406 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.878383    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c7cd73e0-5d9c-46be-9532-5661607b7b39-audit-log\") pod \"metrics-server-76bb975779-mgnrc\" (UID: \"c7cd73e0-5d9c-46be-9532-5661607b7b39\") " pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:16:27.878746 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.878603    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7cd73e0-5d9c-46be-9532-5661607b7b39-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76bb975779-mgnrc\" (UID: \"c7cd73e0-5d9c-46be-9532-5661607b7b39\") " pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:16:27.878964 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.878831    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c7cd73e0-5d9c-46be-9532-5661607b7b39-metrics-server-audit-profiles\") pod \"metrics-server-76bb975779-mgnrc\" (UID: \"c7cd73e0-5d9c-46be-9532-5661607b7b39\") " pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:16:27.880430 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.880411    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c7cd73e0-5d9c-46be-9532-5661607b7b39-secret-metrics-server-client-certs\") pod \"metrics-server-76bb975779-mgnrc\" (UID: \"c7cd73e0-5d9c-46be-9532-5661607b7b39\") " pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:16:27.880674 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.880653    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c7cd73e0-5d9c-46be-9532-5661607b7b39-secret-metrics-server-tls\") pod \"metrics-server-76bb975779-mgnrc\" (UID: \"c7cd73e0-5d9c-46be-9532-5661607b7b39\") " pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:16:27.880713 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.880669    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c7cd73e0-5d9c-46be-9532-5661607b7b39-client-ca-bundle\") pod \"metrics-server-76bb975779-mgnrc\" (UID: \"c7cd73e0-5d9c-46be-9532-5661607b7b39\") " pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:16:27.886157 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.886130    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff6pk\" (UniqueName: \"kubernetes.io/projected/c7cd73e0-5d9c-46be-9532-5661607b7b39-kube-api-access-ff6pk\") pod \"metrics-server-76bb975779-mgnrc\" (UID: \"c7cd73e0-5d9c-46be-9532-5661607b7b39\") " pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:16:27.988899 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:27.988817    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:16:28.273814 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:28.273743    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76bb975779-mgnrc"]
Sep 18 13:16:28.277086 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:16:28.277054    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7cd73e0_5d9c_46be_9532_5661607b7b39.slice/crio-3a2981c12a3e6a6b46edd32ca87f684d12298bebe62d10d763187a880cb9f849 WatchSource:0}: Error finding container 3a2981c12a3e6a6b46edd32ca87f684d12298bebe62d10d763187a880cb9f849: Status 404 returned error can't find the container with id 3a2981c12a3e6a6b46edd32ca87f684d12298bebe62d10d763187a880cb9f849
Sep 18 13:16:28.347963 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:28.347923    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76bb975779-mgnrc" event={"ID":"c7cd73e0-5d9c-46be-9532-5661607b7b39","Type":"ContainerStarted","Data":"3a2981c12a3e6a6b46edd32ca87f684d12298bebe62d10d763187a880cb9f849"}
Sep 18 13:16:29.502198 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.502172    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 13:16:29.505717 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.505697    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.508297 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.508209    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 13:16:29.508297 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.508252    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 13:16:29.508409 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.508260    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 13:16:29.508535 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.508520    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 13:16:29.508772 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.508759    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 13:16:29.508847 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.508830    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 13:16:29.510175 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.510156    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 13:16:29.510268 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.510162    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 13:16:29.510340 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.510323    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 13:16:29.510492 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.510398    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-48dv24abusv02\""
Sep 18 13:16:29.510617 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.510597    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 13:16:29.510693 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.510650    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 13:16:29.510753 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.510713    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cvph8\""
Sep 18 13:16:29.510800 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.510773    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 13:16:29.513844 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.513824    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 13:16:29.515566 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.515552    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 13:16:29.527802 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.527779    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 13:16:29.591097 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.591050    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.591097 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.591094    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.591344 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.591123    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.591344 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.591173    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f597d415-b5d6-48d9-bdc4-601062b835e3-config-out\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.591344 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.591292    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.591344 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.591324    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.591484 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.591345    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.591484 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.591389    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.591484 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.591413    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.591484 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.591474    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f597d415-b5d6-48d9-bdc4-601062b835e3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.591614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.591495    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.591614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.591520    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-config\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.591614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.591542    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.591614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.591580    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.591751 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.591609    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.591751 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.591632    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.591751 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.591647    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxg4n\" (UniqueName: \"kubernetes.io/projected/f597d415-b5d6-48d9-bdc4-601062b835e3-kube-api-access-zxg4n\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.591751 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.591673    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.591751 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.591707    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-web-config\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.591751 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.591749    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.692398 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.692360    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-web-config\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.692398 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.692396    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.692620 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.692415    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.692620 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.692434    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.692620 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.692452    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.692620 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.692476    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f597d415-b5d6-48d9-bdc4-601062b835e3-config-out\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.692620 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.692579    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.692864 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.692638    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.692864 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.692672    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.692864 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.692705    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.692864 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.692730    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.692864 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.692768    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f597d415-b5d6-48d9-bdc4-601062b835e3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.692864 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.692794    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.692864 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.692819    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-config\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.692864 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.692842    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.694073 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.692872    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.694073 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.692918    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.694073 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.692942    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.694073 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.692963    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxg4n\" (UniqueName: \"kubernetes.io/projected/f597d415-b5d6-48d9-bdc4-601062b835e3-kube-api-access-zxg4n\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.694073 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.693064    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.694073 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.693184    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.694073 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.693655    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.698350 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.694435    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.698350 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.695364    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.698350 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.695430    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f597d415-b5d6-48d9-bdc4-601062b835e3-config-out\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.698350 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.695762    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-web-config\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.698350 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.695933    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.698350 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.696425    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.698350 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.696740    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.698350 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.698024    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.698350 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.698066    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.698350 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.698293    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.698350 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.698322    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f597d415-b5d6-48d9-bdc4-601062b835e3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.699160 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.699139    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.699434 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.699412    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.700430 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.700408    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.700811 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.700796    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-config\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.700883 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.700867    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.701487 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.701467    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.705273 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.705254    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxg4n\" (UniqueName: \"kubernetes.io/projected/f597d415-b5d6-48d9-bdc4-601062b835e3-kube-api-access-zxg4n\") pod \"prometheus-k8s-0\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:29.903152 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:29.903122    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:30.193690 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:30.193617    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 13:16:30.196097 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:16:30.196073    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf597d415_b5d6_48d9_bdc4_601062b835e3.slice/crio-55a5043379cc9990a66b8e698db4883d0f051ad2b802487ab6c198d383977299 WatchSource:0}: Error finding container 55a5043379cc9990a66b8e698db4883d0f051ad2b802487ab6c198d383977299: Status 404 returned error can't find the container with id 55a5043379cc9990a66b8e698db4883d0f051ad2b802487ab6c198d383977299
Sep 18 13:16:30.356031 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:30.355994    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f597d415-b5d6-48d9-bdc4-601062b835e3","Type":"ContainerStarted","Data":"55a5043379cc9990a66b8e698db4883d0f051ad2b802487ab6c198d383977299"}
Sep 18 13:16:30.357620 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:30.357597    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76bb975779-mgnrc" event={"ID":"c7cd73e0-5d9c-46be-9532-5661607b7b39","Type":"ContainerStarted","Data":"976a39ac3c476667c3a6515022a36851f7abb840a966de81e2b178c98edcba50"}
Sep 18 13:16:30.374095 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:30.374048    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-76bb975779-mgnrc" podStartSLOduration=2.159559031 podStartE2EDuration="3.374031271s" totalImagesPullingTime="1.21447224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:16:30.373463942 +0000 UTC m=+87.728923769" watchObservedRunningTime="2026-09-18 13:16:30.374031271 +0000 UTC m=+87.729491097"
Sep 18 13:16:31.362659 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:31.362625    2586 generic.go:356] "Generic (PLEG): container finished" podID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerID="aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34" exitCode=0
Sep 18 13:16:31.363049 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:31.362709    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f597d415-b5d6-48d9-bdc4-601062b835e3","Type":"ContainerDied","Data":"aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34"}
Sep 18 13:16:34.287043 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:34.287007    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6645f89c6b-svklk"
Sep 18 13:16:35.385807 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:35.385764    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f597d415-b5d6-48d9-bdc4-601062b835e3","Type":"ContainerStarted","Data":"8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc"}
Sep 18 13:16:35.386278 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:35.385857    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f597d415-b5d6-48d9-bdc4-601062b835e3","Type":"ContainerStarted","Data":"5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56"}
Sep 18 13:16:37.395460 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:37.395422    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f597d415-b5d6-48d9-bdc4-601062b835e3","Type":"ContainerStarted","Data":"f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80"}
Sep 18 13:16:37.395960 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:37.395487    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f597d415-b5d6-48d9-bdc4-601062b835e3","Type":"ContainerStarted","Data":"285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3"}
Sep 18 13:16:37.395960 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:37.395513    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f597d415-b5d6-48d9-bdc4-601062b835e3","Type":"ContainerStarted","Data":"787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579"}
Sep 18 13:16:37.395960 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:37.395536    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f597d415-b5d6-48d9-bdc4-601062b835e3","Type":"ContainerStarted","Data":"563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de"}
Sep 18 13:16:37.430656 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:37.430582    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.573581322 podStartE2EDuration="8.430567637s" totalImagesPullingTime="5.838980507s" totalInitContainerRuntime="18.005808ms" isStatefulPod=false podCreationTimestamp="2026-09-18 13:16:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:16:37.429673737 +0000 UTC m=+94.785133574" watchObservedRunningTime="2026-09-18 13:16:37.430567637 +0000 UTC m=+94.786027475"
Sep 18 13:16:39.904034 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:39.903995    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:16:47.989663 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:47.989565    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:16:47.990066 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:47.989689    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:16:50.312390 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:50.312359    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tmdn8"
Sep 18 13:16:57.947194 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:16:57.947164    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-crvdj_186f3f17-3593-4843-985c-b4bb77d3f28c/serve-healthcheck-canary/0.log"
Sep 18 13:17:07.994611 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:07.994573    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:17:07.999349 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:07.999323    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-76bb975779-mgnrc"
Sep 18 13:17:29.904066 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:29.904027    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:29.923422 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:29.923394    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:30.609484 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:30.609455    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.151937 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.151828    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 13:17:48.152461 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.152135    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="prometheus" containerID="cri-o://5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56" gracePeriod=600
Sep 18 13:17:48.152461 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.152162    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="kube-rbac-proxy" containerID="cri-o://285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3" gracePeriod=600
Sep 18 13:17:48.152461 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.152185    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="thanos-sidecar" containerID="cri-o://563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de" gracePeriod=600
Sep 18 13:17:48.152461 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.152213    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="config-reloader" containerID="cri-o://8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc" gracePeriod=600
Sep 18 13:17:48.152461 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.152182    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="kube-rbac-proxy-web" containerID="cri-o://787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579" gracePeriod=600
Sep 18 13:17:48.152461 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.152234    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="kube-rbac-proxy-thanos" containerID="cri-o://f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80" gracePeriod=600
Sep 18 13:17:48.384594 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.384570    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.543280 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.543179    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-k8s-db\") pod \"f597d415-b5d6-48d9-bdc4-601062b835e3\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") "
Sep 18 13:17:48.543280 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.543230    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-metrics-client-certs\") pod \"f597d415-b5d6-48d9-bdc4-601062b835e3\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") "
Sep 18 13:17:48.543280 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.543254    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-kube-rbac-proxy\") pod \"f597d415-b5d6-48d9-bdc4-601062b835e3\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") "
Sep 18 13:17:48.543545 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.543284    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-configmap-kubelet-serving-ca-bundle\") pod \"f597d415-b5d6-48d9-bdc4-601062b835e3\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") "
Sep 18 13:17:48.543545 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.543308    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f597d415-b5d6-48d9-bdc4-601062b835e3\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") "
Sep 18 13:17:48.543545 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.543332    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-config\") pod \"f597d415-b5d6-48d9-bdc4-601062b835e3\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") "
Sep 18 13:17:48.543545 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.543372    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-grpc-tls\") pod \"f597d415-b5d6-48d9-bdc4-601062b835e3\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") "
Sep 18 13:17:48.543545 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.543442    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-configmap-metrics-client-ca\") pod \"f597d415-b5d6-48d9-bdc4-601062b835e3\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") "
Sep 18 13:17:48.543545 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.543491    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-configmap-serving-certs-ca-bundle\") pod \"f597d415-b5d6-48d9-bdc4-601062b835e3\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") "
Sep 18 13:17:48.543545 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.543520    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f597d415-b5d6-48d9-bdc4-601062b835e3-tls-assets\") pod \"f597d415-b5d6-48d9-bdc4-601062b835e3\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") "
Sep 18 13:17:48.543844 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.543551    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f597d415-b5d6-48d9-bdc4-601062b835e3\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") "
Sep 18 13:17:48.543844 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.543576    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxg4n\" (UniqueName: \"kubernetes.io/projected/f597d415-b5d6-48d9-bdc4-601062b835e3-kube-api-access-zxg4n\") pod \"f597d415-b5d6-48d9-bdc4-601062b835e3\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") "
Sep 18 13:17:48.543844 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.543606    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-prometheus-k8s-tls\") pod \"f597d415-b5d6-48d9-bdc4-601062b835e3\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") "
Sep 18 13:17:48.543844 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.543643    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-k8s-rulefiles-1\") pod \"f597d415-b5d6-48d9-bdc4-601062b835e3\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") "
Sep 18 13:17:48.543844 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.543694    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-trusted-ca-bundle\") pod \"f597d415-b5d6-48d9-bdc4-601062b835e3\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") "
Sep 18 13:17:48.543844 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.543719    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-k8s-rulefiles-2\") pod \"f597d415-b5d6-48d9-bdc4-601062b835e3\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") "
Sep 18 13:17:48.543844 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.543743    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-web-config\") pod \"f597d415-b5d6-48d9-bdc4-601062b835e3\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") "
Sep 18 13:17:48.543844 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.543769    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-k8s-rulefiles-0\") pod \"f597d415-b5d6-48d9-bdc4-601062b835e3\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") "
Sep 18 13:17:48.543844 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.543803    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f597d415-b5d6-48d9-bdc4-601062b835e3-config-out\") pod \"f597d415-b5d6-48d9-bdc4-601062b835e3\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") "
Sep 18 13:17:48.543844 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.543839    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-thanos-prometheus-http-client-file\") pod \"f597d415-b5d6-48d9-bdc4-601062b835e3\" (UID: \"f597d415-b5d6-48d9-bdc4-601062b835e3\") "
Sep 18 13:17:48.546862 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.543763    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "f597d415-b5d6-48d9-bdc4-601062b835e3" (UID: "f597d415-b5d6-48d9-bdc4-601062b835e3"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:17:48.546862 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.544810    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "f597d415-b5d6-48d9-bdc4-601062b835e3" (UID: "f597d415-b5d6-48d9-bdc4-601062b835e3"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:17:48.546862 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.546552    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "f597d415-b5d6-48d9-bdc4-601062b835e3" (UID: "f597d415-b5d6-48d9-bdc4-601062b835e3"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:17:48.546862 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.546659    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "f597d415-b5d6-48d9-bdc4-601062b835e3" (UID: "f597d415-b5d6-48d9-bdc4-601062b835e3"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:17:48.546862 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.546603    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "f597d415-b5d6-48d9-bdc4-601062b835e3" (UID: "f597d415-b5d6-48d9-bdc4-601062b835e3"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:17:48.547248 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.547006    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "f597d415-b5d6-48d9-bdc4-601062b835e3" (UID: "f597d415-b5d6-48d9-bdc4-601062b835e3"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:17:48.547248 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.547088    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "f597d415-b5d6-48d9-bdc4-601062b835e3" (UID: "f597d415-b5d6-48d9-bdc4-601062b835e3"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:17:48.547248 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.547210    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "f597d415-b5d6-48d9-bdc4-601062b835e3" (UID: "f597d415-b5d6-48d9-bdc4-601062b835e3"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:17:48.547407 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.547276    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "f597d415-b5d6-48d9-bdc4-601062b835e3" (UID: "f597d415-b5d6-48d9-bdc4-601062b835e3"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:17:48.547407 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.547290    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f597d415-b5d6-48d9-bdc4-601062b835e3-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "f597d415-b5d6-48d9-bdc4-601062b835e3" (UID: "f597d415-b5d6-48d9-bdc4-601062b835e3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:17:48.547521 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.547402    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "f597d415-b5d6-48d9-bdc4-601062b835e3" (UID: "f597d415-b5d6-48d9-bdc4-601062b835e3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:17:48.547521 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.547464    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "f597d415-b5d6-48d9-bdc4-601062b835e3" (UID: "f597d415-b5d6-48d9-bdc4-601062b835e3"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:17:48.547622 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.547564    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "f597d415-b5d6-48d9-bdc4-601062b835e3" (UID: "f597d415-b5d6-48d9-bdc4-601062b835e3"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:17:48.548072 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.548049    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "f597d415-b5d6-48d9-bdc4-601062b835e3" (UID: "f597d415-b5d6-48d9-bdc4-601062b835e3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:17:48.548176 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.548091    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f597d415-b5d6-48d9-bdc4-601062b835e3-kube-api-access-zxg4n" (OuterVolumeSpecName: "kube-api-access-zxg4n") pod "f597d415-b5d6-48d9-bdc4-601062b835e3" (UID: "f597d415-b5d6-48d9-bdc4-601062b835e3"). InnerVolumeSpecName "kube-api-access-zxg4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:17:48.548706 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.548676    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-config" (OuterVolumeSpecName: "config") pod "f597d415-b5d6-48d9-bdc4-601062b835e3" (UID: "f597d415-b5d6-48d9-bdc4-601062b835e3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:17:48.549008 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.548987    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "f597d415-b5d6-48d9-bdc4-601062b835e3" (UID: "f597d415-b5d6-48d9-bdc4-601062b835e3"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:17:48.549121 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.549095    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "f597d415-b5d6-48d9-bdc4-601062b835e3" (UID: "f597d415-b5d6-48d9-bdc4-601062b835e3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:17:48.549804 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.549779    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f597d415-b5d6-48d9-bdc4-601062b835e3-config-out" (OuterVolumeSpecName: "config-out") pod "f597d415-b5d6-48d9-bdc4-601062b835e3" (UID: "f597d415-b5d6-48d9-bdc4-601062b835e3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:17:48.558618 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.558595    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-web-config" (OuterVolumeSpecName: "web-config") pod "f597d415-b5d6-48d9-bdc4-601062b835e3" (UID: "f597d415-b5d6-48d9-bdc4-601062b835e3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:17:48.644613 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.644577    2586 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-k8s-db\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:17:48.644613 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.644608    2586 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-metrics-client-certs\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:17:48.644613 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.644617    2586 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-kube-rbac-proxy\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:17:48.644837 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.644628    2586 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:17:48.644837 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.644638    2586 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:17:48.644837 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.644648    2586 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:17:48.644837 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.644656    2586 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-grpc-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:17:48.644837 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.644665    2586 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-configmap-metrics-client-ca\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:17:48.644837 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.644672    2586 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:17:48.644837 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.644681    2586 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f597d415-b5d6-48d9-bdc4-601062b835e3-tls-assets\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:17:48.644837 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.644689    2586 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:17:48.644837 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.644697    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxg4n\" (UniqueName: \"kubernetes.io/projected/f597d415-b5d6-48d9-bdc4-601062b835e3-kube-api-access-zxg4n\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:17:48.644837 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.644707    2586 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:17:48.644837 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.644715    2586 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:17:48.644837 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.644723    2586 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:17:48.644837 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.644731    2586 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:17:48.644837 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.644738    2586 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-web-config\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:17:48.644837 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.644746    2586 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f597d415-b5d6-48d9-bdc4-601062b835e3-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:17:48.644837 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.644753    2586 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f597d415-b5d6-48d9-bdc4-601062b835e3-config-out\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:17:48.644837 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.644761    2586 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f597d415-b5d6-48d9-bdc4-601062b835e3-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:17:48.662656 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.662621    2586 generic.go:356] "Generic (PLEG): container finished" podID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerID="f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80" exitCode=0
Sep 18 13:17:48.662656 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.662647    2586 generic.go:356] "Generic (PLEG): container finished" podID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerID="285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3" exitCode=0
Sep 18 13:17:48.662656 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.662656    2586 generic.go:356] "Generic (PLEG): container finished" podID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerID="787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579" exitCode=0
Sep 18 13:17:48.662914 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.662664    2586 generic.go:356] "Generic (PLEG): container finished" podID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerID="563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de" exitCode=0
Sep 18 13:17:48.662914 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.662672    2586 generic.go:356] "Generic (PLEG): container finished" podID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerID="8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc" exitCode=0
Sep 18 13:17:48.662914 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.662679    2586 generic.go:356] "Generic (PLEG): container finished" podID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerID="5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56" exitCode=0
Sep 18 13:17:48.662914 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.662712    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.662914 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.662704    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f597d415-b5d6-48d9-bdc4-601062b835e3","Type":"ContainerDied","Data":"f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80"}
Sep 18 13:17:48.662914 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.662868    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f597d415-b5d6-48d9-bdc4-601062b835e3","Type":"ContainerDied","Data":"285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3"}
Sep 18 13:17:48.663211 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.662931    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f597d415-b5d6-48d9-bdc4-601062b835e3","Type":"ContainerDied","Data":"787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579"}
Sep 18 13:17:48.663211 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.662983    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f597d415-b5d6-48d9-bdc4-601062b835e3","Type":"ContainerDied","Data":"563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de"}
Sep 18 13:17:48.663211 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.662933    2586 scope.go:117] "RemoveContainer" containerID="f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80"
Sep 18 13:17:48.663211 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.663028    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f597d415-b5d6-48d9-bdc4-601062b835e3","Type":"ContainerDied","Data":"8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc"}
Sep 18 13:17:48.663211 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.663067    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f597d415-b5d6-48d9-bdc4-601062b835e3","Type":"ContainerDied","Data":"5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56"}
Sep 18 13:17:48.663211 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.663108    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f597d415-b5d6-48d9-bdc4-601062b835e3","Type":"ContainerDied","Data":"55a5043379cc9990a66b8e698db4883d0f051ad2b802487ab6c198d383977299"}
Sep 18 13:17:48.671948 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.671924    2586 scope.go:117] "RemoveContainer" containerID="285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3"
Sep 18 13:17:48.678750 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.678736    2586 scope.go:117] "RemoveContainer" containerID="787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579"
Sep 18 13:17:48.685175 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.685160    2586 scope.go:117] "RemoveContainer" containerID="563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de"
Sep 18 13:17:48.687171 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.687148    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 13:17:48.692286 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.692268    2586 scope.go:117] "RemoveContainer" containerID="8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc"
Sep 18 13:17:48.695749 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.695731    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 13:17:48.699603 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.699586    2586 scope.go:117] "RemoveContainer" containerID="5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56"
Sep 18 13:17:48.706373 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.706357    2586 scope.go:117] "RemoveContainer" containerID="aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34"
Sep 18 13:17:48.712823 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.712807    2586 scope.go:117] "RemoveContainer" containerID="f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80"
Sep 18 13:17:48.713165 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:17:48.713145    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80\": container with ID starting with f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80 not found: ID does not exist" containerID="f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80"
Sep 18 13:17:48.713232 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.713173    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80"} err="failed to get container status \"f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80\": rpc error: code = NotFound desc = could not find container \"f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80\": container with ID starting with f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80 not found: ID does not exist"
Sep 18 13:17:48.713232 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.713206    2586 scope.go:117] "RemoveContainer" containerID="285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3"
Sep 18 13:17:48.713490 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:17:48.713474    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3\": container with ID starting with 285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3 not found: ID does not exist" containerID="285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3"
Sep 18 13:17:48.713550 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.713493    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3"} err="failed to get container status \"285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3\": rpc error: code = NotFound desc = could not find container \"285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3\": container with ID starting with 285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3 not found: ID does not exist"
Sep 18 13:17:48.713550 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.713511    2586 scope.go:117] "RemoveContainer" containerID="787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579"
Sep 18 13:17:48.713724 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:17:48.713706    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579\": container with ID starting with 787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579 not found: ID does not exist" containerID="787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579"
Sep 18 13:17:48.713765 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.713729    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579"} err="failed to get container status \"787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579\": rpc error: code = NotFound desc = could not find container \"787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579\": container with ID starting with 787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579 not found: ID does not exist"
Sep 18 13:17:48.713765 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.713742    2586 scope.go:117] "RemoveContainer" containerID="563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de"
Sep 18 13:17:48.714092 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:17:48.714072    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de\": container with ID starting with 563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de not found: ID does not exist" containerID="563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de"
Sep 18 13:17:48.714166 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.714097    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de"} err="failed to get container status \"563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de\": rpc error: code = NotFound desc = could not find container \"563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de\": container with ID starting with 563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de not found: ID does not exist"
Sep 18 13:17:48.714166 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.714112    2586 scope.go:117] "RemoveContainer" containerID="8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc"
Sep 18 13:17:48.714370 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:17:48.714354    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc\": container with ID starting with 8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc not found: ID does not exist" containerID="8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc"
Sep 18 13:17:48.714403 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.714376    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc"} err="failed to get container status \"8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc\": rpc error: code = NotFound desc = could not find container \"8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc\": container with ID starting with 8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc not found: ID does not exist"
Sep 18 13:17:48.714403 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.714393    2586 scope.go:117] "RemoveContainer" containerID="5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56"
Sep 18 13:17:48.714646 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:17:48.714629    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56\": container with ID starting with 5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56 not found: ID does not exist" containerID="5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56"
Sep 18 13:17:48.714691 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.714650    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56"} err="failed to get container status \"5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56\": rpc error: code = NotFound desc = could not find container \"5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56\": container with ID starting with 5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56 not found: ID does not exist"
Sep 18 13:17:48.714691 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.714664    2586 scope.go:117] "RemoveContainer" containerID="aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34"
Sep 18 13:17:48.714880 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:17:48.714862    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34\": container with ID starting with aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34 not found: ID does not exist" containerID="aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34"
Sep 18 13:17:48.714942 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.714883    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34"} err="failed to get container status \"aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34\": rpc error: code = NotFound desc = could not find container \"aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34\": container with ID starting with aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34 not found: ID does not exist"
Sep 18 13:17:48.714942 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.714894    2586 scope.go:117] "RemoveContainer" containerID="f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80"
Sep 18 13:17:48.715140 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.715123    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80"} err="failed to get container status \"f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80\": rpc error: code = NotFound desc = could not find container \"f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80\": container with ID starting with f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80 not found: ID does not exist"
Sep 18 13:17:48.715187 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.715140    2586 scope.go:117] "RemoveContainer" containerID="285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3"
Sep 18 13:17:48.715360 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.715340    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3"} err="failed to get container status \"285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3\": rpc error: code = NotFound desc = could not find container \"285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3\": container with ID starting with 285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3 not found: ID does not exist"
Sep 18 13:17:48.715404 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.715360    2586 scope.go:117] "RemoveContainer" containerID="787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579"
Sep 18 13:17:48.715563 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.715548    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579"} err="failed to get container status \"787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579\": rpc error: code = NotFound desc = could not find container \"787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579\": container with ID starting with 787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579 not found: ID does not exist"
Sep 18 13:17:48.715606 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.715563    2586 scope.go:117] "RemoveContainer" containerID="563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de"
Sep 18 13:17:48.715775 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.715756    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de"} err="failed to get container status \"563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de\": rpc error: code = NotFound desc = could not find container \"563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de\": container with ID starting with 563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de not found: ID does not exist"
Sep 18 13:17:48.715841 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.715777    2586 scope.go:117] "RemoveContainer" containerID="8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc"
Sep 18 13:17:48.716015 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.715999    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc"} err="failed to get container status \"8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc\": rpc error: code = NotFound desc = could not find container \"8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc\": container with ID starting with 8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc not found: ID does not exist"
Sep 18 13:17:48.716059 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.716015    2586 scope.go:117] "RemoveContainer" containerID="5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56"
Sep 18 13:17:48.716255 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.716238    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56"} err="failed to get container status \"5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56\": rpc error: code = NotFound desc = could not find container \"5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56\": container with ID starting with 5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56 not found: ID does not exist"
Sep 18 13:17:48.716255 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.716255    2586 scope.go:117] "RemoveContainer" containerID="aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34"
Sep 18 13:17:48.716487 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.716470    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34"} err="failed to get container status \"aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34\": rpc error: code = NotFound desc = could not find container \"aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34\": container with ID starting with aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34 not found: ID does not exist"
Sep 18 13:17:48.716558 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.716489    2586 scope.go:117] "RemoveContainer" containerID="f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80"
Sep 18 13:17:48.716732 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.716715    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80"} err="failed to get container status \"f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80\": rpc error: code = NotFound desc = could not find container \"f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80\": container with ID starting with f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80 not found: ID does not exist"
Sep 18 13:17:48.716775 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.716734    2586 scope.go:117] "RemoveContainer" containerID="285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3"
Sep 18 13:17:48.716965 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.716948    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3"} err="failed to get container status \"285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3\": rpc error: code = NotFound desc = could not find container \"285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3\": container with ID starting with 285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3 not found: ID does not exist"
Sep 18 13:17:48.717031 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.716966    2586 scope.go:117] "RemoveContainer" containerID="787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579"
Sep 18 13:17:48.717213 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.717196    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579"} err="failed to get container status \"787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579\": rpc error: code = NotFound desc = could not find container \"787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579\": container with ID starting with 787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579 not found: ID does not exist"
Sep 18 13:17:48.717276 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.717214    2586 scope.go:117] "RemoveContainer" containerID="563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de"
Sep 18 13:17:48.717442 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.717426    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de"} err="failed to get container status \"563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de\": rpc error: code = NotFound desc = could not find container \"563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de\": container with ID starting with 563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de not found: ID does not exist"
Sep 18 13:17:48.717490 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.717443    2586 scope.go:117] "RemoveContainer" containerID="8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc"
Sep 18 13:17:48.717669 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.717652    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc"} err="failed to get container status \"8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc\": rpc error: code = NotFound desc = could not find container \"8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc\": container with ID starting with 8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc not found: ID does not exist"
Sep 18 13:17:48.717734 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.717671    2586 scope.go:117] "RemoveContainer" containerID="5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56"
Sep 18 13:17:48.717951 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.717933    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56"} err="failed to get container status \"5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56\": rpc error: code = NotFound desc = could not find container \"5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56\": container with ID starting with 5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56 not found: ID does not exist"
Sep 18 13:17:48.718008 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.717951    2586 scope.go:117] "RemoveContainer" containerID="aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34"
Sep 18 13:17:48.718172 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.718155    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34"} err="failed to get container status \"aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34\": rpc error: code = NotFound desc = could not find container \"aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34\": container with ID starting with aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34 not found: ID does not exist"
Sep 18 13:17:48.718215 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.718173    2586 scope.go:117] "RemoveContainer" containerID="f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80"
Sep 18 13:17:48.718397 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.718378    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80"} err="failed to get container status \"f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80\": rpc error: code = NotFound desc = could not find container \"f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80\": container with ID starting with f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80 not found: ID does not exist"
Sep 18 13:17:48.718463 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.718398    2586 scope.go:117] "RemoveContainer" containerID="285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3"
Sep 18 13:17:48.718635 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.718618    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3"} err="failed to get container status \"285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3\": rpc error: code = NotFound desc = could not find container \"285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3\": container with ID starting with 285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3 not found: ID does not exist"
Sep 18 13:17:48.718680 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.718637    2586 scope.go:117] "RemoveContainer" containerID="787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579"
Sep 18 13:17:48.718837 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.718821    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579"} err="failed to get container status \"787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579\": rpc error: code = NotFound desc = could not find container \"787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579\": container with ID starting with 787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579 not found: ID does not exist"
Sep 18 13:17:48.718881 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.718838    2586 scope.go:117] "RemoveContainer" containerID="563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de"
Sep 18 13:17:48.719091 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.719074    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de"} err="failed to get container status \"563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de\": rpc error: code = NotFound desc = could not find container \"563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de\": container with ID starting with 563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de not found: ID does not exist"
Sep 18 13:17:48.719145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.719091    2586 scope.go:117] "RemoveContainer" containerID="8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc"
Sep 18 13:17:48.719329 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.719310    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc"} err="failed to get container status \"8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc\": rpc error: code = NotFound desc = could not find container \"8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc\": container with ID starting with 8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc not found: ID does not exist"
Sep 18 13:17:48.719369 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.719330    2586 scope.go:117] "RemoveContainer" containerID="5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56"
Sep 18 13:17:48.719566 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.719545    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56"} err="failed to get container status \"5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56\": rpc error: code = NotFound desc = could not find container \"5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56\": container with ID starting with 5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56 not found: ID does not exist"
Sep 18 13:17:48.719566 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.719564    2586 scope.go:117] "RemoveContainer" containerID="aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34"
Sep 18 13:17:48.719769 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.719745    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34"} err="failed to get container status \"aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34\": rpc error: code = NotFound desc = could not find container \"aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34\": container with ID starting with aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34 not found: ID does not exist"
Sep 18 13:17:48.719817 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.719770    2586 scope.go:117] "RemoveContainer" containerID="f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80"
Sep 18 13:17:48.720044 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.720026    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80"} err="failed to get container status \"f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80\": rpc error: code = NotFound desc = could not find container \"f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80\": container with ID starting with f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80 not found: ID does not exist"
Sep 18 13:17:48.720112 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.720046    2586 scope.go:117] "RemoveContainer" containerID="285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3"
Sep 18 13:17:48.720295 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.720275    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3"} err="failed to get container status \"285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3\": rpc error: code = NotFound desc = could not find container \"285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3\": container with ID starting with 285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3 not found: ID does not exist"
Sep 18 13:17:48.720348 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.720295    2586 scope.go:117] "RemoveContainer" containerID="787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579"
Sep 18 13:17:48.720532 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.720517    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579"} err="failed to get container status \"787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579\": rpc error: code = NotFound desc = could not find container \"787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579\": container with ID starting with 787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579 not found: ID does not exist"
Sep 18 13:17:48.720575 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.720532    2586 scope.go:117] "RemoveContainer" containerID="563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de"
Sep 18 13:17:48.720746 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.720729    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de"} err="failed to get container status \"563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de\": rpc error: code = NotFound desc = could not find container \"563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de\": container with ID starting with 563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de not found: ID does not exist"
Sep 18 13:17:48.720746 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.720745    2586 scope.go:117] "RemoveContainer" containerID="8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc"
Sep 18 13:17:48.720995 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.720979    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc"} err="failed to get container status \"8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc\": rpc error: code = NotFound desc = could not find container \"8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc\": container with ID starting with 8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc not found: ID does not exist"
Sep 18 13:17:48.721039 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.720995    2586 scope.go:117] "RemoveContainer" containerID="5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56"
Sep 18 13:17:48.721216 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.721201    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56"} err="failed to get container status \"5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56\": rpc error: code = NotFound desc = could not find container \"5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56\": container with ID starting with 5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56 not found: ID does not exist"
Sep 18 13:17:48.721216 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.721216    2586 scope.go:117] "RemoveContainer" containerID="aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34"
Sep 18 13:17:48.721435 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.721416    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34"} err="failed to get container status \"aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34\": rpc error: code = NotFound desc = could not find container \"aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34\": container with ID starting with aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34 not found: ID does not exist"
Sep 18 13:17:48.721480 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.721436    2586 scope.go:117] "RemoveContainer" containerID="f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80"
Sep 18 13:17:48.721657 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.721639    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80"} err="failed to get container status \"f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80\": rpc error: code = NotFound desc = could not find container \"f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80\": container with ID starting with f8f9aef91a960e56ec5658d111dcba89e827dc8dab1202968c0cc2fedcb0df80 not found: ID does not exist"
Sep 18 13:17:48.721709 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.721657    2586 scope.go:117] "RemoveContainer" containerID="285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3"
Sep 18 13:17:48.721884 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.721868    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3"} err="failed to get container status \"285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3\": rpc error: code = NotFound desc = could not find container \"285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3\": container with ID starting with 285bc32b45c016ba618541964445d0cc1873460558783c48d91c2a9fce0b53f3 not found: ID does not exist"
Sep 18 13:17:48.721941 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.721884    2586 scope.go:117] "RemoveContainer" containerID="787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579"
Sep 18 13:17:48.722106 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.722089    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579"} err="failed to get container status \"787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579\": rpc error: code = NotFound desc = could not find container \"787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579\": container with ID starting with 787128e85836796740945f9dd8331a1f9068480df365e920c98ce5b94be94579 not found: ID does not exist"
Sep 18 13:17:48.722149 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.722106    2586 scope.go:117] "RemoveContainer" containerID="563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de"
Sep 18 13:17:48.722314 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.722299    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de"} err="failed to get container status \"563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de\": rpc error: code = NotFound desc = could not find container \"563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de\": container with ID starting with 563549ecad9eb5b8b2f35865b16f63840ffff8470ab5b4a53ed9d662cd0939de not found: ID does not exist"
Sep 18 13:17:48.722377 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.722314    2586 scope.go:117] "RemoveContainer" containerID="8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc"
Sep 18 13:17:48.722537 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.722521    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc"} err="failed to get container status \"8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc\": rpc error: code = NotFound desc = could not find container \"8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc\": container with ID starting with 8e5e452e7aaccd35acb1ae61184c28ba72b21a2d96dbba9ffa51158a9506b4dc not found: ID does not exist"
Sep 18 13:17:48.722537 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.722536    2586 scope.go:117] "RemoveContainer" containerID="5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56"
Sep 18 13:17:48.722707 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.722693    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56"} err="failed to get container status \"5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56\": rpc error: code = NotFound desc = could not find container \"5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56\": container with ID starting with 5b3ee3756efd8dafd47feae671f1d77d6226cba0e6e3f1046659969636c4aa56 not found: ID does not exist"
Sep 18 13:17:48.722746 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.722707    2586 scope.go:117] "RemoveContainer" containerID="aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34"
Sep 18 13:17:48.722956 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.722921    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34"} err="failed to get container status \"aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34\": rpc error: code = NotFound desc = could not find container \"aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34\": container with ID starting with aaefe95a188f98e7b0fb758b289f711b78cb9bd47f67d265fdbd452ec707ad34 not found: ID does not exist"
Sep 18 13:17:48.746471 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.746446    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 13:17:48.746679 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.746668    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="init-config-reloader"
Sep 18 13:17:48.746726 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.746681    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="init-config-reloader"
Sep 18 13:17:48.746726 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.746689    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="kube-rbac-proxy-thanos"
Sep 18 13:17:48.746726 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.746694    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="kube-rbac-proxy-thanos"
Sep 18 13:17:48.746726 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.746703    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="config-reloader"
Sep 18 13:17:48.746726 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.746709    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="config-reloader"
Sep 18 13:17:48.746726 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.746715    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="thanos-sidecar"
Sep 18 13:17:48.746726 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.746721    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="thanos-sidecar"
Sep 18 13:17:48.746726 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.746729    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="kube-rbac-proxy"
Sep 18 13:17:48.746956 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.746734    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="kube-rbac-proxy"
Sep 18 13:17:48.746956 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.746742    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="prometheus"
Sep 18 13:17:48.746956 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.746747    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="prometheus"
Sep 18 13:17:48.746956 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.746752    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="kube-rbac-proxy-web"
Sep 18 13:17:48.746956 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.746757    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="kube-rbac-proxy-web"
Sep 18 13:17:48.746956 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.746797    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="thanos-sidecar"
Sep 18 13:17:48.746956 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.746804    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="config-reloader"
Sep 18 13:17:48.746956 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.746808    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="kube-rbac-proxy-thanos"
Sep 18 13:17:48.746956 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.746813    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="kube-rbac-proxy-web"
Sep 18 13:17:48.746956 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.746818    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="prometheus"
Sep 18 13:17:48.746956 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.746823    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" containerName="kube-rbac-proxy"
Sep 18 13:17:48.750662 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.750646    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.754324 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.754306    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 13:17:48.754713 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.754696    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 13:17:48.754788 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.754710    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 13:17:48.754788 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.754741    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 13:17:48.754894 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.754710    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-48dv24abusv02\""
Sep 18 13:17:48.754894 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.754816    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 13:17:48.754894 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.754800    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 13:17:48.755149 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.755131    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 13:17:48.755235 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.755161    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 13:17:48.755235 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.755187    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 13:17:48.755235 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.755199    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 13:17:48.755235 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.755219    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cvph8\""
Sep 18 13:17:48.755404 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.755161    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 13:17:48.755509 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.755493    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 13:17:48.758215 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.758197    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 13:17:48.761853 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.761832    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 13:17:48.773736 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.773718    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 13:17:48.845766 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.845684    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80708264-fa72-4841-9597-3c902b788ac1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.845928 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.845761    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80708264-fa72-4841-9597-3c902b788ac1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.845928 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.845818    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/80708264-fa72-4841-9597-3c902b788ac1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.845928 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.845850    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/80708264-fa72-4841-9597-3c902b788ac1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.845928 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.845877    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/80708264-fa72-4841-9597-3c902b788ac1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.846149 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.845938    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80708264-fa72-4841-9597-3c902b788ac1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.846149 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.846008    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.846149 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.846042    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/80708264-fa72-4841-9597-3c902b788ac1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.846149 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.846072    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.846149 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.846098    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.846149 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.846134    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80708264-fa72-4841-9597-3c902b788ac1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.846429 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.846157    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-web-config\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.846429 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.846183    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80708264-fa72-4841-9597-3c902b788ac1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.846429 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.846209    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-config\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.846429 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.846255    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.846429 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.846280    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80708264-fa72-4841-9597-3c902b788ac1-config-out\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.846429 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.846305    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.846429 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.846340    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.846429 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.846369    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.846429 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.846398    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86xl4\" (UniqueName: \"kubernetes.io/projected/80708264-fa72-4841-9597-3c902b788ac1-kube-api-access-86xl4\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.848325 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.848299    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f597d415-b5d6-48d9-bdc4-601062b835e3" path="/var/lib/kubelet/pods/f597d415-b5d6-48d9-bdc4-601062b835e3/volumes"
Sep 18 13:17:48.946732 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.946695    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/80708264-fa72-4841-9597-3c902b788ac1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.946732 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.946735    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/80708264-fa72-4841-9597-3c902b788ac1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.947024 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.946761    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80708264-fa72-4841-9597-3c902b788ac1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.947024 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.946793    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.947024 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.946819    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/80708264-fa72-4841-9597-3c902b788ac1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.947024 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.946844    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.947024 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.946867    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.947024 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.946888    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80708264-fa72-4841-9597-3c902b788ac1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.947024 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.946945    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-web-config\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.947024 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.946973    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80708264-fa72-4841-9597-3c902b788ac1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.947024 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.947001    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-config\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.947440 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.947051    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.947440 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.947082    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80708264-fa72-4841-9597-3c902b788ac1-config-out\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.947440 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.947108    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.947440 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.947143    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.947440 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.947174    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.947440 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.947204    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86xl4\" (UniqueName: \"kubernetes.io/projected/80708264-fa72-4841-9597-3c902b788ac1-kube-api-access-86xl4\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.947440 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.947246    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80708264-fa72-4841-9597-3c902b788ac1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.947440 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.947277    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/80708264-fa72-4841-9597-3c902b788ac1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.947832 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.947436    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/80708264-fa72-4841-9597-3c902b788ac1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.947832 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.947282    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80708264-fa72-4841-9597-3c902b788ac1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.947832 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.947733    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/80708264-fa72-4841-9597-3c902b788ac1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.948021 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.947965    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80708264-fa72-4841-9597-3c902b788ac1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.950037 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.948400    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80708264-fa72-4841-9597-3c902b788ac1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.950037 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.948529    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/80708264-fa72-4841-9597-3c902b788ac1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.950037 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.949302    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80708264-fa72-4841-9597-3c902b788ac1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.950037 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.949990    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80708264-fa72-4841-9597-3c902b788ac1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.950425 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.950377    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.951033 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.950975    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.951972 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.951454    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.951972 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.951931    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-web-config\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.952179 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.952057    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-config\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.952179 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.952083    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80708264-fa72-4841-9597-3c902b788ac1-config-out\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.952295 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.952271    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.952503 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.952481    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.952727 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.952710    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.952795 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.952712    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/80708264-fa72-4841-9597-3c902b788ac1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.953129 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.953111    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80708264-fa72-4841-9597-3c902b788ac1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.953449 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.953433    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80708264-fa72-4841-9597-3c902b788ac1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:48.957532 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:48.957509    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86xl4\" (UniqueName: \"kubernetes.io/projected/80708264-fa72-4841-9597-3c902b788ac1-kube-api-access-86xl4\") pod \"prometheus-k8s-0\" (UID: \"80708264-fa72-4841-9597-3c902b788ac1\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:49.059941 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:49.059879    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:17:49.353323 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:49.353284    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 13:17:49.355443 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:17:49.355381    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80708264_fa72_4841_9597_3c902b788ac1.slice/crio-88cba96d96f8956eadab91d851a7a3b4485f6dbf2cd3a2ec9d4d2f83204181a6 WatchSource:0}: Error finding container 88cba96d96f8956eadab91d851a7a3b4485f6dbf2cd3a2ec9d4d2f83204181a6: Status 404 returned error can't find the container with id 88cba96d96f8956eadab91d851a7a3b4485f6dbf2cd3a2ec9d4d2f83204181a6
Sep 18 13:17:49.667705 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:49.667666    2586 generic.go:356] "Generic (PLEG): container finished" podID="80708264-fa72-4841-9597-3c902b788ac1" containerID="708f3d313fbac96c9934d04d55b457b6d059bb15b44c7b8f186ac0c1a004257c" exitCode=0
Sep 18 13:17:49.667875 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:49.667756    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80708264-fa72-4841-9597-3c902b788ac1","Type":"ContainerDied","Data":"708f3d313fbac96c9934d04d55b457b6d059bb15b44c7b8f186ac0c1a004257c"}
Sep 18 13:17:49.667875 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:49.667828    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80708264-fa72-4841-9597-3c902b788ac1","Type":"ContainerStarted","Data":"88cba96d96f8956eadab91d851a7a3b4485f6dbf2cd3a2ec9d4d2f83204181a6"}
Sep 18 13:17:50.675860 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:50.675823    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80708264-fa72-4841-9597-3c902b788ac1","Type":"ContainerStarted","Data":"ea2776c71b119270a8ee1989aa6bcf9162393b3d815894708036ec09dc1783c7"}
Sep 18 13:17:50.676249 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:50.675931    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80708264-fa72-4841-9597-3c902b788ac1","Type":"ContainerStarted","Data":"015b91bd7e91d6af1ded8357f282aa6e739fcf686d58d92b9d9e18ea469e3aae"}
Sep 18 13:17:50.676249 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:50.675975    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80708264-fa72-4841-9597-3c902b788ac1","Type":"ContainerStarted","Data":"a8ff7ec884b3686a9bdc1dda2a66feec9cf687785b950e8b92bd8068b54f5893"}
Sep 18 13:17:50.676249 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:50.676024    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80708264-fa72-4841-9597-3c902b788ac1","Type":"ContainerStarted","Data":"9130a09851745d8b5683e62fd8112e96471d7c49ade8a1098670a6f01366f2ed"}
Sep 18 13:17:50.676249 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:50.676063    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80708264-fa72-4841-9597-3c902b788ac1","Type":"ContainerStarted","Data":"71329946539e1249fcfd021b473c9e1c61e12d449377d852a9daee3677b8cd91"}
Sep 18 13:17:50.676249 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:50.676098    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80708264-fa72-4841-9597-3c902b788ac1","Type":"ContainerStarted","Data":"6bb65c07826481cb35770ad8a5e36d567060c583dc30f6b85dc9dfbfd1e760fb"}
Sep 18 13:17:50.711623 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:50.711575    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.686588972 podStartE2EDuration="2.711555107s" totalImagesPullingTime="0s" totalInitContainerRuntime="24.966135ms" isStatefulPod=false podCreationTimestamp="2026-09-18 13:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:17:50.705838087 +0000 UTC m=+168.061297923" watchObservedRunningTime="2026-09-18 13:17:50.711555107 +0000 UTC m=+168.067014931"
Sep 18 13:17:54.060342 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:17:54.060301    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:18:49.061018 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:18:49.060976    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:18:49.076781 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:18:49.076754    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:18:49.913274 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:18:49.913249    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 13:20:04.749795 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:04.749765    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lcqcw_1cc32441-1189-4113-bf17-2769fa23a747/ovn-acl-logging/0.log"
Sep 18 13:20:04.751559 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:04.751536    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lcqcw_1cc32441-1189-4113-bf17-2769fa23a747/ovn-acl-logging/0.log"
Sep 18 13:20:04.753724 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:04.753706    2586 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 13:20:56.543660 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:56.543575    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw"]
Sep 18 13:20:56.546977 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:56.546956    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw"
Sep 18 13:20:56.549586 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:56.549563    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-krfj5\""
Sep 18 13:20:56.551203 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:56.551185    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 18 13:20:56.551426 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:56.551408    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 18 13:20:56.551502 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:56.551474    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 18 13:20:56.551760 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:56.551713    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 18 13:20:56.565639 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:56.565619    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw"]
Sep 18 13:20:56.597436 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:56.597397    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v74r\" (UniqueName: \"kubernetes.io/projected/52171301-42af-4d64-a163-908f491dd2e1-kube-api-access-2v74r\") pod \"opendatahub-operator-controller-manager-854585bc8d-g56vw\" (UID: \"52171301-42af-4d64-a163-908f491dd2e1\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw"
Sep 18 13:20:56.597594 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:56.597490    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/52171301-42af-4d64-a163-908f491dd2e1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-854585bc8d-g56vw\" (UID: \"52171301-42af-4d64-a163-908f491dd2e1\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw"
Sep 18 13:20:56.597594 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:56.597526    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/52171301-42af-4d64-a163-908f491dd2e1-manifests\") pod \"opendatahub-operator-controller-manager-854585bc8d-g56vw\" (UID: \"52171301-42af-4d64-a163-908f491dd2e1\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw"
Sep 18 13:20:56.597698 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:56.597611    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52171301-42af-4d64-a163-908f491dd2e1-webhook-cert\") pod \"opendatahub-operator-controller-manager-854585bc8d-g56vw\" (UID: \"52171301-42af-4d64-a163-908f491dd2e1\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw"
Sep 18 13:20:56.597746 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:56.597699    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52171301-42af-4d64-a163-908f491dd2e1-tmp\") pod \"opendatahub-operator-controller-manager-854585bc8d-g56vw\" (UID: \"52171301-42af-4d64-a163-908f491dd2e1\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw"
Sep 18 13:20:56.698205 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:56.698168    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/52171301-42af-4d64-a163-908f491dd2e1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-854585bc8d-g56vw\" (UID: \"52171301-42af-4d64-a163-908f491dd2e1\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw"
Sep 18 13:20:56.698205 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:56.698210    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/52171301-42af-4d64-a163-908f491dd2e1-manifests\") pod \"opendatahub-operator-controller-manager-854585bc8d-g56vw\" (UID: \"52171301-42af-4d64-a163-908f491dd2e1\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw"
Sep 18 13:20:56.698451 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:56.698235    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52171301-42af-4d64-a163-908f491dd2e1-webhook-cert\") pod \"opendatahub-operator-controller-manager-854585bc8d-g56vw\" (UID: \"52171301-42af-4d64-a163-908f491dd2e1\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw"
Sep 18 13:20:56.698451 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:56.698277    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52171301-42af-4d64-a163-908f491dd2e1-tmp\") pod \"opendatahub-operator-controller-manager-854585bc8d-g56vw\" (UID: \"52171301-42af-4d64-a163-908f491dd2e1\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw"
Sep 18 13:20:56.698451 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:56.698297    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v74r\" (UniqueName: \"kubernetes.io/projected/52171301-42af-4d64-a163-908f491dd2e1-kube-api-access-2v74r\") pod \"opendatahub-operator-controller-manager-854585bc8d-g56vw\" (UID: \"52171301-42af-4d64-a163-908f491dd2e1\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw"
Sep 18 13:20:56.698637 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:56.698606    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/52171301-42af-4d64-a163-908f491dd2e1-manifests\") pod \"opendatahub-operator-controller-manager-854585bc8d-g56vw\" (UID: \"52171301-42af-4d64-a163-908f491dd2e1\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw"
Sep 18 13:20:56.700740 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:56.700722    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52171301-42af-4d64-a163-908f491dd2e1-tmp\") pod \"opendatahub-operator-controller-manager-854585bc8d-g56vw\" (UID: \"52171301-42af-4d64-a163-908f491dd2e1\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw"
Sep 18 13:20:56.700850 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:56.700795    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/52171301-42af-4d64-a163-908f491dd2e1-webhook-cert\") pod \"opendatahub-operator-controller-manager-854585bc8d-g56vw\" (UID: \"52171301-42af-4d64-a163-908f491dd2e1\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw"
Sep 18 13:20:56.700927 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:56.700851    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/52171301-42af-4d64-a163-908f491dd2e1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-854585bc8d-g56vw\" (UID: \"52171301-42af-4d64-a163-908f491dd2e1\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw"
Sep 18 13:20:56.706595 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:56.706576    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v74r\" (UniqueName: \"kubernetes.io/projected/52171301-42af-4d64-a163-908f491dd2e1-kube-api-access-2v74r\") pod \"opendatahub-operator-controller-manager-854585bc8d-g56vw\" (UID: \"52171301-42af-4d64-a163-908f491dd2e1\") " pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw"
Sep 18 13:20:56.857276 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:56.857202    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw"
Sep 18 13:20:57.173671 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:57.173632    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw"]
Sep 18 13:20:57.180606 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:20:57.180578    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52171301_42af_4d64_a163_908f491dd2e1.slice/crio-3d26ceefbcd51de430f541569b5c84b863a6499e2d4c88a48141eeeef909bdec WatchSource:0}: Error finding container 3d26ceefbcd51de430f541569b5c84b863a6499e2d4c88a48141eeeef909bdec: Status 404 returned error can't find the container with id 3d26ceefbcd51de430f541569b5c84b863a6499e2d4c88a48141eeeef909bdec
Sep 18 13:20:57.182401 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:57.182383    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 13:20:57.337729 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:20:57.337699    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw" event={"ID":"52171301-42af-4d64-a163-908f491dd2e1","Type":"ContainerStarted","Data":"3d26ceefbcd51de430f541569b5c84b863a6499e2d4c88a48141eeeef909bdec"}
Sep 18 13:21:00.352841 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:00.352806    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw" event={"ID":"52171301-42af-4d64-a163-908f491dd2e1","Type":"ContainerStarted","Data":"b35ddf7b8956d4c85d57a30083615f798b7710b95ae91cd99f7abd7f60b541f2"}
Sep 18 13:21:01.357696 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:01.357661    2586 generic.go:356] "Generic (PLEG): container finished" podID="52171301-42af-4d64-a163-908f491dd2e1" containerID="b35ddf7b8956d4c85d57a30083615f798b7710b95ae91cd99f7abd7f60b541f2" exitCode=0
Sep 18 13:21:01.358086 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:01.357700    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw" event={"ID":"52171301-42af-4d64-a163-908f491dd2e1","Type":"ContainerDied","Data":"b35ddf7b8956d4c85d57a30083615f798b7710b95ae91cd99f7abd7f60b541f2"}
Sep 18 13:21:02.363361 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:02.363327    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw" event={"ID":"52171301-42af-4d64-a163-908f491dd2e1","Type":"ContainerStarted","Data":"ebfe6f711d14271a38c74ca038f43ae4f7d180c58ea9b300d63540df447f2486"}
Sep 18 13:21:02.363727 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:02.363425    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw"
Sep 18 13:21:02.391326 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:02.391270    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw" podStartSLOduration=1.9445030330000002 podStartE2EDuration="6.391246736s" totalImagesPullingTime="3.415882029s" totalInitContainerRuntime="1.030861674s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:20:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:21:02.390042115 +0000 UTC m=+359.745501940" watchObservedRunningTime="2026-09-18 13:21:02.391246736 +0000 UTC m=+359.746706563"
Sep 18 13:21:13.369543 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:13.369511    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-854585bc8d-g56vw"
Sep 18 13:21:36.642848 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:36.642812    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-85985676b5-vqt6t"]
Sep 18 13:21:36.647551 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:36.647535    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85985676b5-vqt6t"
Sep 18 13:21:36.650468 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:36.650442    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 18 13:21:36.650575 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:36.650442    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 13:21:36.651320 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:36.651299    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 13:21:36.651501 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:36.651484    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 13:21:36.651561 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:36.651520    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 13:21:36.656975 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:36.656956    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-n2mgq\""
Sep 18 13:21:36.670773 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:36.670740    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85985676b5-vqt6t"]
Sep 18 13:21:36.742196 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:36.742156    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4846c6d7-ea37-4392-9c57-ec4b348b5684-metrics-cert\") pod \"lws-controller-manager-85985676b5-vqt6t\" (UID: \"4846c6d7-ea37-4392-9c57-ec4b348b5684\") " pod="openshift-lws-operator/lws-controller-manager-85985676b5-vqt6t"
Sep 18 13:21:36.742370 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:36.742202    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl2bn\" (UniqueName: \"kubernetes.io/projected/4846c6d7-ea37-4392-9c57-ec4b348b5684-kube-api-access-kl2bn\") pod \"lws-controller-manager-85985676b5-vqt6t\" (UID: \"4846c6d7-ea37-4392-9c57-ec4b348b5684\") " pod="openshift-lws-operator/lws-controller-manager-85985676b5-vqt6t"
Sep 18 13:21:36.742370 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:36.742250    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4846c6d7-ea37-4392-9c57-ec4b348b5684-cert\") pod \"lws-controller-manager-85985676b5-vqt6t\" (UID: \"4846c6d7-ea37-4392-9c57-ec4b348b5684\") " pod="openshift-lws-operator/lws-controller-manager-85985676b5-vqt6t"
Sep 18 13:21:36.742370 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:36.742319    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4846c6d7-ea37-4392-9c57-ec4b348b5684-manager-config\") pod \"lws-controller-manager-85985676b5-vqt6t\" (UID: \"4846c6d7-ea37-4392-9c57-ec4b348b5684\") " pod="openshift-lws-operator/lws-controller-manager-85985676b5-vqt6t"
Sep 18 13:21:36.843397 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:36.843357    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4846c6d7-ea37-4392-9c57-ec4b348b5684-cert\") pod \"lws-controller-manager-85985676b5-vqt6t\" (UID: \"4846c6d7-ea37-4392-9c57-ec4b348b5684\") " pod="openshift-lws-operator/lws-controller-manager-85985676b5-vqt6t"
Sep 18 13:21:36.843584 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:36.843420    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4846c6d7-ea37-4392-9c57-ec4b348b5684-manager-config\") pod \"lws-controller-manager-85985676b5-vqt6t\" (UID: \"4846c6d7-ea37-4392-9c57-ec4b348b5684\") " pod="openshift-lws-operator/lws-controller-manager-85985676b5-vqt6t"
Sep 18 13:21:36.843584 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:36.843454    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4846c6d7-ea37-4392-9c57-ec4b348b5684-metrics-cert\") pod \"lws-controller-manager-85985676b5-vqt6t\" (UID: \"4846c6d7-ea37-4392-9c57-ec4b348b5684\") " pod="openshift-lws-operator/lws-controller-manager-85985676b5-vqt6t"
Sep 18 13:21:36.843584 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:36.843477    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl2bn\" (UniqueName: \"kubernetes.io/projected/4846c6d7-ea37-4392-9c57-ec4b348b5684-kube-api-access-kl2bn\") pod \"lws-controller-manager-85985676b5-vqt6t\" (UID: \"4846c6d7-ea37-4392-9c57-ec4b348b5684\") " pod="openshift-lws-operator/lws-controller-manager-85985676b5-vqt6t"
Sep 18 13:21:36.844248 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:36.844226    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4846c6d7-ea37-4392-9c57-ec4b348b5684-manager-config\") pod \"lws-controller-manager-85985676b5-vqt6t\" (UID: \"4846c6d7-ea37-4392-9c57-ec4b348b5684\") " pod="openshift-lws-operator/lws-controller-manager-85985676b5-vqt6t"
Sep 18 13:21:36.846232 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:36.846204    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4846c6d7-ea37-4392-9c57-ec4b348b5684-metrics-cert\") pod \"lws-controller-manager-85985676b5-vqt6t\" (UID: \"4846c6d7-ea37-4392-9c57-ec4b348b5684\") " pod="openshift-lws-operator/lws-controller-manager-85985676b5-vqt6t"
Sep 18 13:21:36.846340 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:36.846259    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4846c6d7-ea37-4392-9c57-ec4b348b5684-cert\") pod \"lws-controller-manager-85985676b5-vqt6t\" (UID: \"4846c6d7-ea37-4392-9c57-ec4b348b5684\") " pod="openshift-lws-operator/lws-controller-manager-85985676b5-vqt6t"
Sep 18 13:21:36.858388 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:36.858359    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl2bn\" (UniqueName: \"kubernetes.io/projected/4846c6d7-ea37-4392-9c57-ec4b348b5684-kube-api-access-kl2bn\") pod \"lws-controller-manager-85985676b5-vqt6t\" (UID: \"4846c6d7-ea37-4392-9c57-ec4b348b5684\") " pod="openshift-lws-operator/lws-controller-manager-85985676b5-vqt6t"
Sep 18 13:21:36.956857 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:36.956777    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-85985676b5-vqt6t"
Sep 18 13:21:37.235114 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:37.235038    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-85985676b5-vqt6t"]
Sep 18 13:21:37.238041 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:21:37.238019    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4846c6d7_ea37_4392_9c57_ec4b348b5684.slice/crio-0f00ba68e867ed0a198d3cdd5f4dc5251f60bc1a00e99057da171f2df4826e49 WatchSource:0}: Error finding container 0f00ba68e867ed0a198d3cdd5f4dc5251f60bc1a00e99057da171f2df4826e49: Status 404 returned error can't find the container with id 0f00ba68e867ed0a198d3cdd5f4dc5251f60bc1a00e99057da171f2df4826e49
Sep 18 13:21:37.497838 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:37.497752    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85985676b5-vqt6t" event={"ID":"4846c6d7-ea37-4392-9c57-ec4b348b5684","Type":"ContainerStarted","Data":"0f00ba68e867ed0a198d3cdd5f4dc5251f60bc1a00e99057da171f2df4826e49"}
Sep 18 13:21:39.036094 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:39.036051    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-ws7zp"]
Sep 18 13:21:39.045313 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:39.045290    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ws7zp"
Sep 18 13:21:39.066644 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:39.066602    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f0b1bf8-da36-4c37-8600-b5a16a85246f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-ws7zp\" (UID: \"3f0b1bf8-da36-4c37-8600-b5a16a85246f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ws7zp"
Sep 18 13:21:39.066783 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:39.066668    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwccd\" (UniqueName: \"kubernetes.io/projected/3f0b1bf8-da36-4c37-8600-b5a16a85246f-kube-api-access-wwccd\") pod \"llmisvc-controller-manager-777c6bd7d5-ws7zp\" (UID: \"3f0b1bf8-da36-4c37-8600-b5a16a85246f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ws7zp"
Sep 18 13:21:39.082028 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:39.081958    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-ws7zp"]
Sep 18 13:21:39.092060 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:39.091744    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-bbqrv\""
Sep 18 13:21:39.093012 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:39.092808    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 18 13:21:39.167175 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:39.167133    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f0b1bf8-da36-4c37-8600-b5a16a85246f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-ws7zp\" (UID: \"3f0b1bf8-da36-4c37-8600-b5a16a85246f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ws7zp"
Sep 18 13:21:39.167369 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:39.167201    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwccd\" (UniqueName: \"kubernetes.io/projected/3f0b1bf8-da36-4c37-8600-b5a16a85246f-kube-api-access-wwccd\") pod \"llmisvc-controller-manager-777c6bd7d5-ws7zp\" (UID: \"3f0b1bf8-da36-4c37-8600-b5a16a85246f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ws7zp"
Sep 18 13:21:39.167529 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:21:39.167489    2586 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 18 13:21:39.167646 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:21:39.167563    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f0b1bf8-da36-4c37-8600-b5a16a85246f-cert podName:3f0b1bf8-da36-4c37-8600-b5a16a85246f nodeName:}" failed. No retries permitted until 2026-09-18 13:21:39.667542215 +0000 UTC m=+397.023002033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3f0b1bf8-da36-4c37-8600-b5a16a85246f-cert") pod "llmisvc-controller-manager-777c6bd7d5-ws7zp" (UID: "3f0b1bf8-da36-4c37-8600-b5a16a85246f") : secret "llmisvc-webhook-server-cert" not found
Sep 18 13:21:39.191030 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:39.191000    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwccd\" (UniqueName: \"kubernetes.io/projected/3f0b1bf8-da36-4c37-8600-b5a16a85246f-kube-api-access-wwccd\") pod \"llmisvc-controller-manager-777c6bd7d5-ws7zp\" (UID: \"3f0b1bf8-da36-4c37-8600-b5a16a85246f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ws7zp"
Sep 18 13:21:39.670234 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:39.670190    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f0b1bf8-da36-4c37-8600-b5a16a85246f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-ws7zp\" (UID: \"3f0b1bf8-da36-4c37-8600-b5a16a85246f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ws7zp"
Sep 18 13:21:39.672848 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:39.672817    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f0b1bf8-da36-4c37-8600-b5a16a85246f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-ws7zp\" (UID: \"3f0b1bf8-da36-4c37-8600-b5a16a85246f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ws7zp"
Sep 18 13:21:39.957829 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:39.957731    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ws7zp"
Sep 18 13:21:40.268198 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:40.268103    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-ws7zp"]
Sep 18 13:21:40.272492 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:21:40.272461    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3f0b1bf8_da36_4c37_8600_b5a16a85246f.slice/crio-46eed7e40298e3287dc36276f6149b56ae550cc917874e1a208a7a207dabb71e WatchSource:0}: Error finding container 46eed7e40298e3287dc36276f6149b56ae550cc917874e1a208a7a207dabb71e: Status 404 returned error can't find the container with id 46eed7e40298e3287dc36276f6149b56ae550cc917874e1a208a7a207dabb71e
Sep 18 13:21:40.513800 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:40.513765    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-85985676b5-vqt6t" event={"ID":"4846c6d7-ea37-4392-9c57-ec4b348b5684","Type":"ContainerStarted","Data":"30054563e95955848eafd6a89c1ad7caeb113df63210c275af8500b7bb593b4e"}
Sep 18 13:21:40.514012 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:40.513874    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-85985676b5-vqt6t"
Sep 18 13:21:40.515009 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:40.514988    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ws7zp" event={"ID":"3f0b1bf8-da36-4c37-8600-b5a16a85246f","Type":"ContainerStarted","Data":"46eed7e40298e3287dc36276f6149b56ae550cc917874e1a208a7a207dabb71e"}
Sep 18 13:21:43.529635 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:43.529551    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ws7zp" event={"ID":"3f0b1bf8-da36-4c37-8600-b5a16a85246f","Type":"ContainerStarted","Data":"10bd71e02fbc0632282896fdc9ad2d45196804600eb9f6d87288ab775a637a89"}
Sep 18 13:21:43.530093 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:43.529726    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ws7zp"
Sep 18 13:21:43.595476 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:43.595426    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-85985676b5-vqt6t" podStartSLOduration=5.150387207 podStartE2EDuration="7.595408871s" totalImagesPullingTime="2.445021664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:21:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:21:40.551278776 +0000 UTC m=+397.906738612" watchObservedRunningTime="2026-09-18 13:21:43.595408871 +0000 UTC m=+400.950868696"
Sep 18 13:21:51.521795 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:51.521767    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-85985676b5-vqt6t"
Sep 18 13:21:51.546621 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:21:51.546568    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ws7zp" podStartSLOduration=10.653036312 podStartE2EDuration="13.546550042s" totalImagesPullingTime="2.89351373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:21:43.595562885 +0000 UTC m=+400.951022707" watchObservedRunningTime="2026-09-18 13:21:51.546550042 +0000 UTC m=+408.902009922"
Sep 18 13:22:14.536431 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:22:14.536394    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ws7zp"
Sep 18 13:23:20.525637 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:20.525590    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg"]
Sep 18 13:23:20.528965 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:20.528948    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg"
Sep 18 13:23:20.533757 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:20.533738    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 13:23:20.536195 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:20.536177    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sbkdk\""
Sep 18 13:23:20.536646 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:20.536630    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 13:23:20.564107 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:20.564080    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg"]
Sep 18 13:23:20.620209 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:20.620177    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2l7p\" (UniqueName: \"kubernetes.io/projected/ad7f15d2-7e4f-422c-816a-d35e644d573b-kube-api-access-v2l7p\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7hnrg\" (UID: \"ad7f15d2-7e4f-422c-816a-d35e644d573b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg"
Sep 18 13:23:20.620344 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:20.620231    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad7f15d2-7e4f-422c-816a-d35e644d573b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7hnrg\" (UID: \"ad7f15d2-7e4f-422c-816a-d35e644d573b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg"
Sep 18 13:23:20.721424 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:20.721390    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2l7p\" (UniqueName: \"kubernetes.io/projected/ad7f15d2-7e4f-422c-816a-d35e644d573b-kube-api-access-v2l7p\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7hnrg\" (UID: \"ad7f15d2-7e4f-422c-816a-d35e644d573b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg"
Sep 18 13:23:20.721575 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:20.721453    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad7f15d2-7e4f-422c-816a-d35e644d573b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7hnrg\" (UID: \"ad7f15d2-7e4f-422c-816a-d35e644d573b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg"
Sep 18 13:23:20.721844 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:20.721828    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad7f15d2-7e4f-422c-816a-d35e644d573b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7hnrg\" (UID: \"ad7f15d2-7e4f-422c-816a-d35e644d573b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg"
Sep 18 13:23:20.740529 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:20.740500    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2l7p\" (UniqueName: \"kubernetes.io/projected/ad7f15d2-7e4f-422c-816a-d35e644d573b-kube-api-access-v2l7p\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7hnrg\" (UID: \"ad7f15d2-7e4f-422c-816a-d35e644d573b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg"
Sep 18 13:23:20.838835 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:20.838736    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg"
Sep 18 13:23:21.154681 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:21.154652    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg"]
Sep 18 13:23:21.156078 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:23:21.156050    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad7f15d2_7e4f_422c_816a_d35e644d573b.slice/crio-4ec28290aec1ab92766fc2040bf585ef3738f4c014a60011d4e84beb4ad0498e WatchSource:0}: Error finding container 4ec28290aec1ab92766fc2040bf585ef3738f4c014a60011d4e84beb4ad0498e: Status 404 returned error can't find the container with id 4ec28290aec1ab92766fc2040bf585ef3738f4c014a60011d4e84beb4ad0498e
Sep 18 13:23:21.892892 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:21.892839    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg" event={"ID":"ad7f15d2-7e4f-422c-816a-d35e644d573b","Type":"ContainerStarted","Data":"4ec28290aec1ab92766fc2040bf585ef3738f4c014a60011d4e84beb4ad0498e"}
Sep 18 13:23:25.913407 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:25.913372    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg" event={"ID":"ad7f15d2-7e4f-422c-816a-d35e644d573b","Type":"ContainerStarted","Data":"c68f2889f1bfbb70d939fc952f178820f9af01e4f41124a80e84ed7f5aee0b07"}
Sep 18 13:23:25.913781 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:25.913498    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg"
Sep 18 13:23:25.942890 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:25.942834    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg" podStartSLOduration=2.144178497 podStartE2EDuration="5.942817834s" totalImagesPullingTime="3.798639337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:23:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:23:25.941263249 +0000 UTC m=+503.296723073" watchObservedRunningTime="2026-09-18 13:23:25.942817834 +0000 UTC m=+503.298277659"
Sep 18 13:23:36.920118 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:36.920090    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg"
Sep 18 13:23:59.294480 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:59.294402    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:23:59.298615 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:59.298591    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-9stmh"
Sep 18 13:23:59.301722 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:59.301699    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mvlcr\""
Sep 18 13:23:59.301832 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:59.301699    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 18 13:23:59.310099 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:59.310074    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:23:59.347025 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:59.346992    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:23:59.352443 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:59.352419    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/70be07c6-1642-4f67-baea-3a49b4006e07-config-file\") pod \"limitador-limitador-84994f7ffc-9stmh\" (UID: \"70be07c6-1642-4f67-baea-3a49b4006e07\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-9stmh"
Sep 18 13:23:59.352566 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:59.352494    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75dbg\" (UniqueName: \"kubernetes.io/projected/70be07c6-1642-4f67-baea-3a49b4006e07-kube-api-access-75dbg\") pod \"limitador-limitador-84994f7ffc-9stmh\" (UID: \"70be07c6-1642-4f67-baea-3a49b4006e07\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-9stmh"
Sep 18 13:23:59.453458 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:59.453418    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75dbg\" (UniqueName: \"kubernetes.io/projected/70be07c6-1642-4f67-baea-3a49b4006e07-kube-api-access-75dbg\") pod \"limitador-limitador-84994f7ffc-9stmh\" (UID: \"70be07c6-1642-4f67-baea-3a49b4006e07\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-9stmh"
Sep 18 13:23:59.453631 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:59.453465    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/70be07c6-1642-4f67-baea-3a49b4006e07-config-file\") pod \"limitador-limitador-84994f7ffc-9stmh\" (UID: \"70be07c6-1642-4f67-baea-3a49b4006e07\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-9stmh"
Sep 18 13:23:59.454176 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:59.454154    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/70be07c6-1642-4f67-baea-3a49b4006e07-config-file\") pod \"limitador-limitador-84994f7ffc-9stmh\" (UID: \"70be07c6-1642-4f67-baea-3a49b4006e07\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-9stmh"
Sep 18 13:23:59.461871 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:59.461851    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75dbg\" (UniqueName: \"kubernetes.io/projected/70be07c6-1642-4f67-baea-3a49b4006e07-kube-api-access-75dbg\") pod \"limitador-limitador-84994f7ffc-9stmh\" (UID: \"70be07c6-1642-4f67-baea-3a49b4006e07\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-9stmh"
Sep 18 13:23:59.607791 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:59.607705    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-9stmh"
Sep 18 13:23:59.886723 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:23:59.886690    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:23:59.889866 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:23:59.889836    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70be07c6_1642_4f67_baea_3a49b4006e07.slice/crio-fcd4986c01cc720cc1f2dd2545971ce7cdb2c12052c0edd12fbaa05339b1aad9 WatchSource:0}: Error finding container fcd4986c01cc720cc1f2dd2545971ce7cdb2c12052c0edd12fbaa05339b1aad9: Status 404 returned error can't find the container with id fcd4986c01cc720cc1f2dd2545971ce7cdb2c12052c0edd12fbaa05339b1aad9
Sep 18 13:24:00.060468 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:24:00.060432    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-9stmh" event={"ID":"70be07c6-1642-4f67-baea-3a49b4006e07","Type":"ContainerStarted","Data":"fcd4986c01cc720cc1f2dd2545971ce7cdb2c12052c0edd12fbaa05339b1aad9"}
Sep 18 13:24:01.066737 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:24:01.066701    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-9stmh" event={"ID":"70be07c6-1642-4f67-baea-3a49b4006e07","Type":"ContainerStarted","Data":"39717381833e7ac683a9c8229043d61b8d2b02df15018298d8d0cec60fa24ad9"}
Sep 18 13:24:01.067106 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:24:01.066829    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-9stmh"
Sep 18 13:24:01.083085 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:24:01.083039    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-9stmh" podStartSLOduration=1.017680033 podStartE2EDuration="2.083024135s" totalImagesPullingTime="1.065344102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:23:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:24:01.081888653 +0000 UTC m=+538.437348478" watchObservedRunningTime="2026-09-18 13:24:01.083024135 +0000 UTC m=+538.438483960"
Sep 18 13:24:12.076090 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:24:12.076061    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-9stmh"
Sep 18 13:25:04.778200 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:04.778172    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lcqcw_1cc32441-1189-4113-bf17-2769fa23a747/ovn-acl-logging/0.log"
Sep 18 13:25:04.779100 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:04.779073    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lcqcw_1cc32441-1189-4113-bf17-2769fa23a747/ovn-acl-logging/0.log"
Sep 18 13:25:13.763275 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:13.763238    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:25:35.985659 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:35.985573    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:25:49.683044 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.683014    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"]
Sep 18 13:25:49.686091 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.686075    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"
Sep 18 13:25:49.689056 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.689032    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fjbgv\""
Sep 18 13:25:49.689644 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.689626    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 18 13:25:49.689717 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.689631    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 18 13:25:49.689717 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.689662    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 18 13:25:49.701412 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.701388    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"]
Sep 18 13:25:49.780627 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.780597    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9414bb6-2317-4e59-bbd8-8ebc5096749c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6pxr8\" (UID: \"a9414bb6-2317-4e59-bbd8-8ebc5096749c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"
Sep 18 13:25:49.780786 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.780638    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp5h8\" (UniqueName: \"kubernetes.io/projected/a9414bb6-2317-4e59-bbd8-8ebc5096749c-kube-api-access-rp5h8\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6pxr8\" (UID: \"a9414bb6-2317-4e59-bbd8-8ebc5096749c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"
Sep 18 13:25:49.780786 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.780665    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9414bb6-2317-4e59-bbd8-8ebc5096749c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6pxr8\" (UID: \"a9414bb6-2317-4e59-bbd8-8ebc5096749c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"
Sep 18 13:25:49.780786 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.780752    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9414bb6-2317-4e59-bbd8-8ebc5096749c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6pxr8\" (UID: \"a9414bb6-2317-4e59-bbd8-8ebc5096749c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"
Sep 18 13:25:49.780920 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.780793    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9414bb6-2317-4e59-bbd8-8ebc5096749c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6pxr8\" (UID: \"a9414bb6-2317-4e59-bbd8-8ebc5096749c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"
Sep 18 13:25:49.780920 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.780815    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9414bb6-2317-4e59-bbd8-8ebc5096749c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6pxr8\" (UID: \"a9414bb6-2317-4e59-bbd8-8ebc5096749c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"
Sep 18 13:25:49.881779 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.881741    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9414bb6-2317-4e59-bbd8-8ebc5096749c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6pxr8\" (UID: \"a9414bb6-2317-4e59-bbd8-8ebc5096749c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"
Sep 18 13:25:49.881779 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.881779    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9414bb6-2317-4e59-bbd8-8ebc5096749c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6pxr8\" (UID: \"a9414bb6-2317-4e59-bbd8-8ebc5096749c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"
Sep 18 13:25:49.882009 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.881832    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9414bb6-2317-4e59-bbd8-8ebc5096749c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6pxr8\" (UID: \"a9414bb6-2317-4e59-bbd8-8ebc5096749c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"
Sep 18 13:25:49.882009 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.881849    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp5h8\" (UniqueName: \"kubernetes.io/projected/a9414bb6-2317-4e59-bbd8-8ebc5096749c-kube-api-access-rp5h8\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6pxr8\" (UID: \"a9414bb6-2317-4e59-bbd8-8ebc5096749c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"
Sep 18 13:25:49.882009 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.881870    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9414bb6-2317-4e59-bbd8-8ebc5096749c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6pxr8\" (UID: \"a9414bb6-2317-4e59-bbd8-8ebc5096749c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"
Sep 18 13:25:49.882009 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.881940    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9414bb6-2317-4e59-bbd8-8ebc5096749c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6pxr8\" (UID: \"a9414bb6-2317-4e59-bbd8-8ebc5096749c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"
Sep 18 13:25:49.882236 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.882212    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9414bb6-2317-4e59-bbd8-8ebc5096749c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6pxr8\" (UID: \"a9414bb6-2317-4e59-bbd8-8ebc5096749c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"
Sep 18 13:25:49.882322 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.882224    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9414bb6-2317-4e59-bbd8-8ebc5096749c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6pxr8\" (UID: \"a9414bb6-2317-4e59-bbd8-8ebc5096749c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"
Sep 18 13:25:49.882377 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.882317    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9414bb6-2317-4e59-bbd8-8ebc5096749c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6pxr8\" (UID: \"a9414bb6-2317-4e59-bbd8-8ebc5096749c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"
Sep 18 13:25:49.884484 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.884463    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9414bb6-2317-4e59-bbd8-8ebc5096749c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6pxr8\" (UID: \"a9414bb6-2317-4e59-bbd8-8ebc5096749c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"
Sep 18 13:25:49.884575 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.884559    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9414bb6-2317-4e59-bbd8-8ebc5096749c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6pxr8\" (UID: \"a9414bb6-2317-4e59-bbd8-8ebc5096749c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"
Sep 18 13:25:49.890599 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.890577    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp5h8\" (UniqueName: \"kubernetes.io/projected/a9414bb6-2317-4e59-bbd8-8ebc5096749c-kube-api-access-rp5h8\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6pxr8\" (UID: \"a9414bb6-2317-4e59-bbd8-8ebc5096749c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"
Sep 18 13:25:49.995986 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:49.995882    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"
Sep 18 13:25:50.269802 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:50.269723    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"]
Sep 18 13:25:50.272732 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:25:50.272693    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9414bb6_2317_4e59_bbd8_8ebc5096749c.slice/crio-5b8e5edb0b55cb8c128a617e355ea072319451bb8c69a053359eeca687bb0b2a WatchSource:0}: Error finding container 5b8e5edb0b55cb8c128a617e355ea072319451bb8c69a053359eeca687bb0b2a: Status 404 returned error can't find the container with id 5b8e5edb0b55cb8c128a617e355ea072319451bb8c69a053359eeca687bb0b2a
Sep 18 13:25:50.531080 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:50.530959    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8" event={"ID":"a9414bb6-2317-4e59-bbd8-8ebc5096749c","Type":"ContainerStarted","Data":"5b8e5edb0b55cb8c128a617e355ea072319451bb8c69a053359eeca687bb0b2a"}
Sep 18 13:25:50.647164 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:50.647130    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:25:51.536300 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:51.536259    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8" event={"ID":"a9414bb6-2317-4e59-bbd8-8ebc5096749c","Type":"ContainerStarted","Data":"11685259fd9dcd40d6da3a462d083c3c4c965060df5469c7c9f6640ddeef8beb"}
Sep 18 13:25:51.558185 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:51.558004    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8" podStartSLOduration=1.5267771460000001 podStartE2EDuration="2.557980169s" totalImagesPullingTime="1.031203023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:25:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:25:51.557536036 +0000 UTC m=+648.912995862" watchObservedRunningTime="2026-09-18 13:25:51.557980169 +0000 UTC m=+648.913439995"
Sep 18 13:25:58.252102 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:58.252055    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"]
Sep 18 13:25:58.255790 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:58.255768    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"
Sep 18 13:25:58.257803 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:58.257776    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 18 13:25:58.268314 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:58.268292    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"]
Sep 18 13:25:58.354395 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:58.354360    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/755a9ecb-c5f9-4c8a-82b8-eaef5989ab70-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv\" (UID: \"755a9ecb-c5f9-4c8a-82b8-eaef5989ab70\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"
Sep 18 13:25:58.354556 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:58.354403    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm9g4\" (UniqueName: \"kubernetes.io/projected/755a9ecb-c5f9-4c8a-82b8-eaef5989ab70-kube-api-access-qm9g4\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv\" (UID: \"755a9ecb-c5f9-4c8a-82b8-eaef5989ab70\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"
Sep 18 13:25:58.354556 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:58.354483    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/755a9ecb-c5f9-4c8a-82b8-eaef5989ab70-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv\" (UID: \"755a9ecb-c5f9-4c8a-82b8-eaef5989ab70\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"
Sep 18 13:25:58.354556 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:58.354508    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/755a9ecb-c5f9-4c8a-82b8-eaef5989ab70-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv\" (UID: \"755a9ecb-c5f9-4c8a-82b8-eaef5989ab70\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"
Sep 18 13:25:58.354556 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:58.354531    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/755a9ecb-c5f9-4c8a-82b8-eaef5989ab70-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv\" (UID: \"755a9ecb-c5f9-4c8a-82b8-eaef5989ab70\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"
Sep 18 13:25:58.354688 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:58.354558    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/755a9ecb-c5f9-4c8a-82b8-eaef5989ab70-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv\" (UID: \"755a9ecb-c5f9-4c8a-82b8-eaef5989ab70\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"
Sep 18 13:25:58.455753 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:58.455718    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/755a9ecb-c5f9-4c8a-82b8-eaef5989ab70-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv\" (UID: \"755a9ecb-c5f9-4c8a-82b8-eaef5989ab70\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"
Sep 18 13:25:58.455931 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:58.455768    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/755a9ecb-c5f9-4c8a-82b8-eaef5989ab70-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv\" (UID: \"755a9ecb-c5f9-4c8a-82b8-eaef5989ab70\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"
Sep 18 13:25:58.455931 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:58.455894    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm9g4\" (UniqueName: \"kubernetes.io/projected/755a9ecb-c5f9-4c8a-82b8-eaef5989ab70-kube-api-access-qm9g4\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv\" (UID: \"755a9ecb-c5f9-4c8a-82b8-eaef5989ab70\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"
Sep 18 13:25:58.456022 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:58.456006    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/755a9ecb-c5f9-4c8a-82b8-eaef5989ab70-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv\" (UID: \"755a9ecb-c5f9-4c8a-82b8-eaef5989ab70\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"
Sep 18 13:25:58.456058 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:58.456035    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/755a9ecb-c5f9-4c8a-82b8-eaef5989ab70-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv\" (UID: \"755a9ecb-c5f9-4c8a-82b8-eaef5989ab70\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"
Sep 18 13:25:58.456090 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:58.456070    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/755a9ecb-c5f9-4c8a-82b8-eaef5989ab70-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv\" (UID: \"755a9ecb-c5f9-4c8a-82b8-eaef5989ab70\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"
Sep 18 13:25:58.456214 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:58.456187    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/755a9ecb-c5f9-4c8a-82b8-eaef5989ab70-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv\" (UID: \"755a9ecb-c5f9-4c8a-82b8-eaef5989ab70\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"
Sep 18 13:25:58.456354 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:58.456335    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/755a9ecb-c5f9-4c8a-82b8-eaef5989ab70-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv\" (UID: \"755a9ecb-c5f9-4c8a-82b8-eaef5989ab70\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"
Sep 18 13:25:58.456448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:58.456432    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/755a9ecb-c5f9-4c8a-82b8-eaef5989ab70-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv\" (UID: \"755a9ecb-c5f9-4c8a-82b8-eaef5989ab70\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"
Sep 18 13:25:58.458260 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:58.458240    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/755a9ecb-c5f9-4c8a-82b8-eaef5989ab70-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv\" (UID: \"755a9ecb-c5f9-4c8a-82b8-eaef5989ab70\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"
Sep 18 13:25:58.458362 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:58.458345    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/755a9ecb-c5f9-4c8a-82b8-eaef5989ab70-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv\" (UID: \"755a9ecb-c5f9-4c8a-82b8-eaef5989ab70\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"
Sep 18 13:25:58.463298 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:58.463274    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm9g4\" (UniqueName: \"kubernetes.io/projected/755a9ecb-c5f9-4c8a-82b8-eaef5989ab70-kube-api-access-qm9g4\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv\" (UID: \"755a9ecb-c5f9-4c8a-82b8-eaef5989ab70\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"
Sep 18 13:25:58.565845 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:58.565776    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"
Sep 18 13:25:58.838481 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:58.838400    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"]
Sep 18 13:25:58.841737 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:25:58.841708    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod755a9ecb_c5f9_4c8a_82b8_eaef5989ab70.slice/crio-32ded8ff7e4be5fb8cc0a2a574f5406200526b02c80790e7ff7fcc1aca1d0c4b WatchSource:0}: Error finding container 32ded8ff7e4be5fb8cc0a2a574f5406200526b02c80790e7ff7fcc1aca1d0c4b: Status 404 returned error can't find the container with id 32ded8ff7e4be5fb8cc0a2a574f5406200526b02c80790e7ff7fcc1aca1d0c4b
Sep 18 13:25:58.843142 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:58.843122    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 13:25:59.369592 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:59.369559    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:25:59.572082 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:59.572040    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv" event={"ID":"755a9ecb-c5f9-4c8a-82b8-eaef5989ab70","Type":"ContainerStarted","Data":"6e61d1dd19978937c0aca20725d0bdf1598c5af79d2bfb590e370a06234f6394"}
Sep 18 13:25:59.572252 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:59.572096    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv" event={"ID":"755a9ecb-c5f9-4c8a-82b8-eaef5989ab70","Type":"ContainerStarted","Data":"32ded8ff7e4be5fb8cc0a2a574f5406200526b02c80790e7ff7fcc1aca1d0c4b"}
Sep 18 13:25:59.589772 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:59.589725    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv" podStartSLOduration=1.416476715 podStartE2EDuration="1.58970958s" totalImagesPullingTime="173.232865ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:25:59.588516073 +0000 UTC m=+656.943975910" watchObservedRunningTime="2026-09-18 13:25:59.58970958 +0000 UTC m=+656.945169405"
Sep 18 13:25:59.996822 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:59.996784    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"
Sep 18 13:25:59.997029 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:25:59.996854    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"
Sep 18 13:26:00.010628 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:00.010597    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"
Sep 18 13:26:00.587133 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:00.587105    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6pxr8"
Sep 18 13:26:08.566997 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:08.566955    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"
Sep 18 13:26:08.567391 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:08.567251    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"
Sep 18 13:26:08.579670 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:08.579648    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"
Sep 18 13:26:08.621020 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:08.620992    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv"
Sep 18 13:26:18.570048 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:18.570016    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:26:34.167031 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:34.166996    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"]
Sep 18 13:26:34.172701 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:34.172675    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"
Sep 18 13:26:34.178249 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:34.178147    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 18 13:26:34.194801 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:34.194769    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"]
Sep 18 13:26:34.272615 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:34.272580    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9b485d0-4b37-4766-8d31-02c8fda334c2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q\" (UID: \"c9b485d0-4b37-4766-8d31-02c8fda334c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"
Sep 18 13:26:34.272778 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:34.272619    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt8sc\" (UniqueName: \"kubernetes.io/projected/c9b485d0-4b37-4766-8d31-02c8fda334c2-kube-api-access-kt8sc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q\" (UID: \"c9b485d0-4b37-4766-8d31-02c8fda334c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"
Sep 18 13:26:34.272778 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:34.272679    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9b485d0-4b37-4766-8d31-02c8fda334c2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q\" (UID: \"c9b485d0-4b37-4766-8d31-02c8fda334c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"
Sep 18 13:26:34.272778 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:34.272711    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9b485d0-4b37-4766-8d31-02c8fda334c2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q\" (UID: \"c9b485d0-4b37-4766-8d31-02c8fda334c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"
Sep 18 13:26:34.272778 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:34.272751    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9b485d0-4b37-4766-8d31-02c8fda334c2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q\" (UID: \"c9b485d0-4b37-4766-8d31-02c8fda334c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"
Sep 18 13:26:34.272943 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:34.272779    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9b485d0-4b37-4766-8d31-02c8fda334c2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q\" (UID: \"c9b485d0-4b37-4766-8d31-02c8fda334c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"
Sep 18 13:26:34.373381 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:34.373344    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9b485d0-4b37-4766-8d31-02c8fda334c2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q\" (UID: \"c9b485d0-4b37-4766-8d31-02c8fda334c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"
Sep 18 13:26:34.373575 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:34.373406    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9b485d0-4b37-4766-8d31-02c8fda334c2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q\" (UID: \"c9b485d0-4b37-4766-8d31-02c8fda334c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"
Sep 18 13:26:34.373575 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:34.373449    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9b485d0-4b37-4766-8d31-02c8fda334c2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q\" (UID: \"c9b485d0-4b37-4766-8d31-02c8fda334c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"
Sep 18 13:26:34.373575 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:34.373483    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt8sc\" (UniqueName: \"kubernetes.io/projected/c9b485d0-4b37-4766-8d31-02c8fda334c2-kube-api-access-kt8sc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q\" (UID: \"c9b485d0-4b37-4766-8d31-02c8fda334c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"
Sep 18 13:26:34.373575 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:34.373527    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9b485d0-4b37-4766-8d31-02c8fda334c2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q\" (UID: \"c9b485d0-4b37-4766-8d31-02c8fda334c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"
Sep 18 13:26:34.373575 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:34.373573    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9b485d0-4b37-4766-8d31-02c8fda334c2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q\" (UID: \"c9b485d0-4b37-4766-8d31-02c8fda334c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"
Sep 18 13:26:34.374030 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:34.374003    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9b485d0-4b37-4766-8d31-02c8fda334c2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q\" (UID: \"c9b485d0-4b37-4766-8d31-02c8fda334c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"
Sep 18 13:26:34.374281 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:34.374260    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9b485d0-4b37-4766-8d31-02c8fda334c2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q\" (UID: \"c9b485d0-4b37-4766-8d31-02c8fda334c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"
Sep 18 13:26:34.374281 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:34.374276    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9b485d0-4b37-4766-8d31-02c8fda334c2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q\" (UID: \"c9b485d0-4b37-4766-8d31-02c8fda334c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"
Sep 18 13:26:34.376195 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:34.376175    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9b485d0-4b37-4766-8d31-02c8fda334c2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q\" (UID: \"c9b485d0-4b37-4766-8d31-02c8fda334c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"
Sep 18 13:26:34.376497 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:34.376475    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9b485d0-4b37-4766-8d31-02c8fda334c2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q\" (UID: \"c9b485d0-4b37-4766-8d31-02c8fda334c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"
Sep 18 13:26:34.388227 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:34.388201    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt8sc\" (UniqueName: \"kubernetes.io/projected/c9b485d0-4b37-4766-8d31-02c8fda334c2-kube-api-access-kt8sc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q\" (UID: \"c9b485d0-4b37-4766-8d31-02c8fda334c2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"
Sep 18 13:26:34.483395 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:34.483306    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"
Sep 18 13:26:34.758530 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:34.758450    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"]
Sep 18 13:26:34.761572 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:26:34.761543    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b485d0_4b37_4766_8d31_02c8fda334c2.slice/crio-ab464ff879e098589e2cb12611dc957d5b87f84db947d54f0f0cb995da8b615d WatchSource:0}: Error finding container ab464ff879e098589e2cb12611dc957d5b87f84db947d54f0f0cb995da8b615d: Status 404 returned error can't find the container with id ab464ff879e098589e2cb12611dc957d5b87f84db947d54f0f0cb995da8b615d
Sep 18 13:26:35.722208 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:35.722165    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q" event={"ID":"c9b485d0-4b37-4766-8d31-02c8fda334c2","Type":"ContainerStarted","Data":"f2e0b3240787b58cba89e67a89173c62b1802d0ffb710872218ac69ef41c58b0"}
Sep 18 13:26:35.722602 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:35.722246    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q" event={"ID":"c9b485d0-4b37-4766-8d31-02c8fda334c2","Type":"ContainerStarted","Data":"ab464ff879e098589e2cb12611dc957d5b87f84db947d54f0f0cb995da8b615d"}
Sep 18 13:26:35.741461 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:35.741282    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q" podStartSLOduration=1.552507967 podStartE2EDuration="1.741263396s" totalImagesPullingTime="188.755429ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:26:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:26:35.740761652 +0000 UTC m=+693.096221476" watchObservedRunningTime="2026-09-18 13:26:35.741263396 +0000 UTC m=+693.096723250"
Sep 18 13:26:35.871229 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:35.871189    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:26:44.483460 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:44.483418    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"
Sep 18 13:26:44.483943 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:44.483493    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"
Sep 18 13:26:44.495941 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:44.495916    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"
Sep 18 13:26:44.777430 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:44.777312    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q"
Sep 18 13:26:46.164274 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:46.164241    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"]
Sep 18 13:26:46.167817 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:46.167799    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"
Sep 18 13:26:46.170639 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:46.170610    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 18 13:26:46.183981 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:46.183960    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"]
Sep 18 13:26:46.279608 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:46.279572    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw8pn\" (UniqueName: \"kubernetes.io/projected/3676d2ca-51a1-4436-967e-af28a0e41be1-kube-api-access-bw8pn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7z2lm\" (UID: \"3676d2ca-51a1-4436-967e-af28a0e41be1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"
Sep 18 13:26:46.279608 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:46.279609    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3676d2ca-51a1-4436-967e-af28a0e41be1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7z2lm\" (UID: \"3676d2ca-51a1-4436-967e-af28a0e41be1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"
Sep 18 13:26:46.279802 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:46.279626    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3676d2ca-51a1-4436-967e-af28a0e41be1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7z2lm\" (UID: \"3676d2ca-51a1-4436-967e-af28a0e41be1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"
Sep 18 13:26:46.279802 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:46.279699    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3676d2ca-51a1-4436-967e-af28a0e41be1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7z2lm\" (UID: \"3676d2ca-51a1-4436-967e-af28a0e41be1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"
Sep 18 13:26:46.279802 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:46.279728    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3676d2ca-51a1-4436-967e-af28a0e41be1-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7z2lm\" (UID: \"3676d2ca-51a1-4436-967e-af28a0e41be1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"
Sep 18 13:26:46.279802 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:46.279743    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3676d2ca-51a1-4436-967e-af28a0e41be1-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7z2lm\" (UID: \"3676d2ca-51a1-4436-967e-af28a0e41be1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"
Sep 18 13:26:46.381086 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:46.381047    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3676d2ca-51a1-4436-967e-af28a0e41be1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7z2lm\" (UID: \"3676d2ca-51a1-4436-967e-af28a0e41be1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"
Sep 18 13:26:46.381086 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:46.381086    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3676d2ca-51a1-4436-967e-af28a0e41be1-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7z2lm\" (UID: \"3676d2ca-51a1-4436-967e-af28a0e41be1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"
Sep 18 13:26:46.381306 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:46.381102    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3676d2ca-51a1-4436-967e-af28a0e41be1-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7z2lm\" (UID: \"3676d2ca-51a1-4436-967e-af28a0e41be1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"
Sep 18 13:26:46.381306 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:46.381182    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw8pn\" (UniqueName: \"kubernetes.io/projected/3676d2ca-51a1-4436-967e-af28a0e41be1-kube-api-access-bw8pn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7z2lm\" (UID: \"3676d2ca-51a1-4436-967e-af28a0e41be1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"
Sep 18 13:26:46.381306 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:46.381209    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3676d2ca-51a1-4436-967e-af28a0e41be1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7z2lm\" (UID: \"3676d2ca-51a1-4436-967e-af28a0e41be1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"
Sep 18 13:26:46.381306 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:46.381228    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3676d2ca-51a1-4436-967e-af28a0e41be1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7z2lm\" (UID: \"3676d2ca-51a1-4436-967e-af28a0e41be1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"
Sep 18 13:26:46.381598 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:46.381562    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3676d2ca-51a1-4436-967e-af28a0e41be1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7z2lm\" (UID: \"3676d2ca-51a1-4436-967e-af28a0e41be1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"
Sep 18 13:26:46.381706 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:46.381595    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3676d2ca-51a1-4436-967e-af28a0e41be1-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7z2lm\" (UID: \"3676d2ca-51a1-4436-967e-af28a0e41be1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"
Sep 18 13:26:46.381760 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:46.381739    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3676d2ca-51a1-4436-967e-af28a0e41be1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7z2lm\" (UID: \"3676d2ca-51a1-4436-967e-af28a0e41be1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"
Sep 18 13:26:46.383550 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:46.383531    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3676d2ca-51a1-4436-967e-af28a0e41be1-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7z2lm\" (UID: \"3676d2ca-51a1-4436-967e-af28a0e41be1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"
Sep 18 13:26:46.383812 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:46.383795    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3676d2ca-51a1-4436-967e-af28a0e41be1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7z2lm\" (UID: \"3676d2ca-51a1-4436-967e-af28a0e41be1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"
Sep 18 13:26:46.390189 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:46.390165    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw8pn\" (UniqueName: \"kubernetes.io/projected/3676d2ca-51a1-4436-967e-af28a0e41be1-kube-api-access-bw8pn\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-7z2lm\" (UID: \"3676d2ca-51a1-4436-967e-af28a0e41be1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"
Sep 18 13:26:46.478828 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:46.478748    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"
Sep 18 13:26:46.754065 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:46.753984    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"]
Sep 18 13:26:46.757399 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:26:46.757363    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3676d2ca_51a1_4436_967e_af28a0e41be1.slice/crio-136c6f40d2ca8abf325e8c9edfb0d7ac7e8f169d854d98a4d2ab03ec7fcf51c9 WatchSource:0}: Error finding container 136c6f40d2ca8abf325e8c9edfb0d7ac7e8f169d854d98a4d2ab03ec7fcf51c9: Status 404 returned error can't find the container with id 136c6f40d2ca8abf325e8c9edfb0d7ac7e8f169d854d98a4d2ab03ec7fcf51c9
Sep 18 13:26:46.776058 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:46.776020    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm" event={"ID":"3676d2ca-51a1-4436-967e-af28a0e41be1","Type":"ContainerStarted","Data":"136c6f40d2ca8abf325e8c9edfb0d7ac7e8f169d854d98a4d2ab03ec7fcf51c9"}
Sep 18 13:26:47.781842 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:47.781792    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm" event={"ID":"3676d2ca-51a1-4436-967e-af28a0e41be1","Type":"ContainerStarted","Data":"e2048d856a5e9215518cc9c6d6756db71be23d4268f76a471b29d1190b6dd716"}
Sep 18 13:26:47.803181 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:47.803123    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm" podStartSLOduration=1.608586627 podStartE2EDuration="1.803103069s" totalImagesPullingTime="194.516442ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:26:47.801156915 +0000 UTC m=+705.156616741" watchObservedRunningTime="2026-09-18 13:26:47.803103069 +0000 UTC m=+705.158562898"
Sep 18 13:26:48.500446 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:48.500415    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:26:56.081631 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:56.081590    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:26:56.478998 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:56.478958    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"
Sep 18 13:26:56.479156 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:56.479030    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"
Sep 18 13:26:56.491593 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:56.491569    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"
Sep 18 13:26:56.832053 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:26:56.831980    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-7z2lm"
Sep 18 13:28:52.586149 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:52.586111    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:28:54.262290 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:54.262255    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7478-5c0a6c"]
Sep 18 13:28:54.265328 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:54.265307    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-5c0a6c"
Sep 18 13:28:54.267298 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:54.267279    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ncnrt\""
Sep 18 13:28:54.272866 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:54.272847    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7478-5c0a6c"]
Sep 18 13:28:54.320669 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:54.320626    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl76j\" (UniqueName: \"kubernetes.io/projected/b1e5d759-6921-42bf-8150-d54b82f3a2ba-kube-api-access-bl76j\") pod \"test-curl-7478-5c0a6c\" (UID: \"b1e5d759-6921-42bf-8150-d54b82f3a2ba\") " pod="opendatahub/test-curl-7478-5c0a6c"
Sep 18 13:28:54.348392 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:54.348364    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2"]
Sep 18 13:28:54.351657 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:54.351638    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2"
Sep 18 13:28:54.353627 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:54.353600    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 13:28:54.353740 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:54.353652    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 13:28:54.353740 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:54.353660    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-34aeca-dockercfg-8rfp7\""
Sep 18 13:28:54.353740 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:54.353720    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-34aeca\""
Sep 18 13:28:54.364574 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:54.364550    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2"]
Sep 18 13:28:54.421090 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:54.421055    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz4r9\" (UniqueName: \"kubernetes.io/projected/27af73d7-5496-42a7-8519-ce4724b2fb9b-kube-api-access-wz4r9\") pod \"payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2\" (UID: \"27af73d7-5496-42a7-8519-ce4724b2fb9b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2"
Sep 18 13:28:54.421256 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:54.421106    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27af73d7-5496-42a7-8519-ce4724b2fb9b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2\" (UID: \"27af73d7-5496-42a7-8519-ce4724b2fb9b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2"
Sep 18 13:28:54.421256 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:54.421134    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl76j\" (UniqueName: \"kubernetes.io/projected/b1e5d759-6921-42bf-8150-d54b82f3a2ba-kube-api-access-bl76j\") pod \"test-curl-7478-5c0a6c\" (UID: \"b1e5d759-6921-42bf-8150-d54b82f3a2ba\") " pod="opendatahub/test-curl-7478-5c0a6c"
Sep 18 13:28:54.440249 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:54.440222    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl76j\" (UniqueName: \"kubernetes.io/projected/b1e5d759-6921-42bf-8150-d54b82f3a2ba-kube-api-access-bl76j\") pod \"test-curl-7478-5c0a6c\" (UID: \"b1e5d759-6921-42bf-8150-d54b82f3a2ba\") " pod="opendatahub/test-curl-7478-5c0a6c"
Sep 18 13:28:54.522185 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:54.522095    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz4r9\" (UniqueName: \"kubernetes.io/projected/27af73d7-5496-42a7-8519-ce4724b2fb9b-kube-api-access-wz4r9\") pod \"payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2\" (UID: \"27af73d7-5496-42a7-8519-ce4724b2fb9b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2"
Sep 18 13:28:54.522185 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:54.522166    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27af73d7-5496-42a7-8519-ce4724b2fb9b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2\" (UID: \"27af73d7-5496-42a7-8519-ce4724b2fb9b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2"
Sep 18 13:28:54.522851 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:54.522826    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27af73d7-5496-42a7-8519-ce4724b2fb9b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2\" (UID: \"27af73d7-5496-42a7-8519-ce4724b2fb9b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2"
Sep 18 13:28:54.529787 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:54.529755    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz4r9\" (UniqueName: \"kubernetes.io/projected/27af73d7-5496-42a7-8519-ce4724b2fb9b-kube-api-access-wz4r9\") pod \"payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2\" (UID: \"27af73d7-5496-42a7-8519-ce4724b2fb9b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2"
Sep 18 13:28:54.575073 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:54.575035    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-5c0a6c"
Sep 18 13:28:54.661576 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:54.661505    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2"
Sep 18 13:28:54.851359 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:54.851290    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7478-5c0a6c"]
Sep 18 13:28:54.854561 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:28:54.854520    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1e5d759_6921_42bf_8150_d54b82f3a2ba.slice/crio-0a156129f7cdd0161eef1fffd3b5e7a8425874a4d12835fcfc3d76098f78de89 WatchSource:0}: Error finding container 0a156129f7cdd0161eef1fffd3b5e7a8425874a4d12835fcfc3d76098f78de89: Status 404 returned error can't find the container with id 0a156129f7cdd0161eef1fffd3b5e7a8425874a4d12835fcfc3d76098f78de89
Sep 18 13:28:54.949585 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:54.949555    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2"]
Sep 18 13:28:54.952622 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:28:54.952592    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27af73d7_5496_42a7_8519_ce4724b2fb9b.slice/crio-608b92e32820c21e8cf66fa7e5eaeab91b04f0f486a4349f5296d35cc260ed79 WatchSource:0}: Error finding container 608b92e32820c21e8cf66fa7e5eaeab91b04f0f486a4349f5296d35cc260ed79: Status 404 returned error can't find the container with id 608b92e32820c21e8cf66fa7e5eaeab91b04f0f486a4349f5296d35cc260ed79
Sep 18 13:28:55.340960 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:55.340928    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2" event={"ID":"27af73d7-5496-42a7-8519-ce4724b2fb9b","Type":"ContainerStarted","Data":"608b92e32820c21e8cf66fa7e5eaeab91b04f0f486a4349f5296d35cc260ed79"}
Sep 18 13:28:55.342150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:55.342128    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7478-5c0a6c" event={"ID":"b1e5d759-6921-42bf-8150-d54b82f3a2ba","Type":"ContainerStarted","Data":"0a156129f7cdd0161eef1fffd3b5e7a8425874a4d12835fcfc3d76098f78de89"}
Sep 18 13:28:55.723007 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:55.722973    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw"]
Sep 18 13:28:55.726944 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:55.726915    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng"]
Sep 18 13:28:55.727088 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:55.727028    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw"
Sep 18 13:28:55.730164 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:55.730143    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-951469\""
Sep 18 13:28:55.730435 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:55.730413    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng"
Sep 18 13:28:55.730813 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:55.730782    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-951469-dockercfg-g9wbt\""
Sep 18 13:28:55.740609 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:55.740584    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw"]
Sep 18 13:28:55.753175 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:55.749237    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng"]
Sep 18 13:28:55.833961 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:55.833890    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz7sk\" (UniqueName: \"kubernetes.io/projected/b8d4efe1-5e52-4f7d-8d17-22715cd25947-kube-api-access-qz7sk\") pod \"payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng\" (UID: \"b8d4efe1-5e52-4f7d-8d17-22715cd25947\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng"
Sep 18 13:28:55.834180 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:55.834022    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw\" (UID: \"3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw"
Sep 18 13:28:55.834180 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:55.834131    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96lv8\" (UniqueName: \"kubernetes.io/projected/3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3-kube-api-access-96lv8\") pod \"payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw\" (UID: \"3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw"
Sep 18 13:28:55.834322 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:55.834204    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d4efe1-5e52-4f7d-8d17-22715cd25947-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng\" (UID: \"b8d4efe1-5e52-4f7d-8d17-22715cd25947\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng"
Sep 18 13:28:55.934768 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:55.934725    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d4efe1-5e52-4f7d-8d17-22715cd25947-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng\" (UID: \"b8d4efe1-5e52-4f7d-8d17-22715cd25947\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng"
Sep 18 13:28:55.934960 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:55.934794    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz7sk\" (UniqueName: \"kubernetes.io/projected/b8d4efe1-5e52-4f7d-8d17-22715cd25947-kube-api-access-qz7sk\") pod \"payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng\" (UID: \"b8d4efe1-5e52-4f7d-8d17-22715cd25947\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng"
Sep 18 13:28:55.934960 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:55.934848    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw\" (UID: \"3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw"
Sep 18 13:28:55.934960 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:55.934951    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96lv8\" (UniqueName: \"kubernetes.io/projected/3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3-kube-api-access-96lv8\") pod \"payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw\" (UID: \"3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw"
Sep 18 13:28:55.936490 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:55.935649    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d4efe1-5e52-4f7d-8d17-22715cd25947-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng\" (UID: \"b8d4efe1-5e52-4f7d-8d17-22715cd25947\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng"
Sep 18 13:28:55.936490 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:55.936447    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw\" (UID: \"3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw"
Sep 18 13:28:55.949393 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:55.949362    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz7sk\" (UniqueName: \"kubernetes.io/projected/b8d4efe1-5e52-4f7d-8d17-22715cd25947-kube-api-access-qz7sk\") pod \"payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng\" (UID: \"b8d4efe1-5e52-4f7d-8d17-22715cd25947\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng"
Sep 18 13:28:55.949523 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:55.949476    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96lv8\" (UniqueName: \"kubernetes.io/projected/3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3-kube-api-access-96lv8\") pod \"payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw\" (UID: \"3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw"
Sep 18 13:28:56.048193 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:56.048104    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw"
Sep 18 13:28:56.056353 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:56.056314    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng"
Sep 18 13:28:56.366358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:56.366292    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw"]
Sep 18 13:28:56.369663 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:28:56.369629    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9a1ca8_e65e_4d6b_b4fc_9ee6c4ec4ad3.slice/crio-f52e07b1cd3f2b8e7ba774769d992eee99903debf252a728fa8419fa2648435a WatchSource:0}: Error finding container f52e07b1cd3f2b8e7ba774769d992eee99903debf252a728fa8419fa2648435a: Status 404 returned error can't find the container with id f52e07b1cd3f2b8e7ba774769d992eee99903debf252a728fa8419fa2648435a
Sep 18 13:28:56.381846 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:56.381817    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng"]
Sep 18 13:28:56.385217 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:28:56.385189    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d4efe1_5e52_4f7d_8d17_22715cd25947.slice/crio-90bbc90b6165c489562cbf2c97d96cbd09ea3f214d8c32ecc9e3f0ed41405fc6 WatchSource:0}: Error finding container 90bbc90b6165c489562cbf2c97d96cbd09ea3f214d8c32ecc9e3f0ed41405fc6: Status 404 returned error can't find the container with id 90bbc90b6165c489562cbf2c97d96cbd09ea3f214d8c32ecc9e3f0ed41405fc6
Sep 18 13:28:56.791154 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:56.791120    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d"]
Sep 18 13:28:56.795693 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:56.795671    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d"
Sep 18 13:28:56.797740 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:56.797704    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-5be8a3\""
Sep 18 13:28:56.797891 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:56.797875    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-5be8a3-dockercfg-lz2jb\""
Sep 18 13:28:56.807081 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:56.807026    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d"]
Sep 18 13:28:56.843222 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:56.843191    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b21c3d-49a1-4fb5-a925-71ee3ff2904f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d\" (UID: \"01b21c3d-49a1-4fb5-a925-71ee3ff2904f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d"
Sep 18 13:28:56.843386 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:56.843272    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr9g8\" (UniqueName: \"kubernetes.io/projected/01b21c3d-49a1-4fb5-a925-71ee3ff2904f-kube-api-access-lr9g8\") pod \"payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d\" (UID: \"01b21c3d-49a1-4fb5-a925-71ee3ff2904f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d"
Sep 18 13:28:56.944272 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:56.944232    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b21c3d-49a1-4fb5-a925-71ee3ff2904f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d\" (UID: \"01b21c3d-49a1-4fb5-a925-71ee3ff2904f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d"
Sep 18 13:28:56.944457 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:56.944333    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr9g8\" (UniqueName: \"kubernetes.io/projected/01b21c3d-49a1-4fb5-a925-71ee3ff2904f-kube-api-access-lr9g8\") pod \"payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d\" (UID: \"01b21c3d-49a1-4fb5-a925-71ee3ff2904f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d"
Sep 18 13:28:56.945072 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:56.945047    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b21c3d-49a1-4fb5-a925-71ee3ff2904f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d\" (UID: \"01b21c3d-49a1-4fb5-a925-71ee3ff2904f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d"
Sep 18 13:28:56.954937 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:56.954893    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr9g8\" (UniqueName: \"kubernetes.io/projected/01b21c3d-49a1-4fb5-a925-71ee3ff2904f-kube-api-access-lr9g8\") pod \"payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d\" (UID: \"01b21c3d-49a1-4fb5-a925-71ee3ff2904f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d"
Sep 18 13:28:57.110246 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:57.110163    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d"
Sep 18 13:28:57.356822 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:57.356783    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng" event={"ID":"b8d4efe1-5e52-4f7d-8d17-22715cd25947","Type":"ContainerStarted","Data":"90bbc90b6165c489562cbf2c97d96cbd09ea3f214d8c32ecc9e3f0ed41405fc6"}
Sep 18 13:28:57.358396 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:57.358368    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw" event={"ID":"3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3","Type":"ContainerStarted","Data":"f52e07b1cd3f2b8e7ba774769d992eee99903debf252a728fa8419fa2648435a"}
Sep 18 13:28:58.638620 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:58.638540    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d"]
Sep 18 13:28:58.641537 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:28:58.641495    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b21c3d_49a1_4fb5_a925_71ee3ff2904f.slice/crio-6e2dfe88b126b7d4c65a40c9198168341fadc88bd1d35d7d700caec6c7cbb003 WatchSource:0}: Error finding container 6e2dfe88b126b7d4c65a40c9198168341fadc88bd1d35d7d700caec6c7cbb003: Status 404 returned error can't find the container with id 6e2dfe88b126b7d4c65a40c9198168341fadc88bd1d35d7d700caec6c7cbb003
Sep 18 13:28:59.369503 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:59.369465    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2" event={"ID":"27af73d7-5496-42a7-8519-ce4724b2fb9b","Type":"ContainerStarted","Data":"7f37821c53a8601ff1a13b69743313732fa38221083d272ca8d4ebc82af2cc9a"}
Sep 18 13:28:59.369701 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:59.369581    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2"
Sep 18 13:28:59.371109 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:59.371087    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng" event={"ID":"b8d4efe1-5e52-4f7d-8d17-22715cd25947","Type":"ContainerStarted","Data":"69c32c1e132a4fd53d62ca5a5b5d3a05fd468bc190b0bb0eb251e710cfc6b420"}
Sep 18 13:28:59.371228 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:59.371178    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng"
Sep 18 13:28:59.372721 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:59.372700    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw" event={"ID":"3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3","Type":"ContainerStarted","Data":"0b40f7e1b819c757b49cb357ab6334240455213259de4a7b560eb886aa5dadbd"}
Sep 18 13:28:59.372833 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:59.372791    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw"
Sep 18 13:28:59.374274 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:59.374254    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7478-5c0a6c" event={"ID":"b1e5d759-6921-42bf-8150-d54b82f3a2ba","Type":"ContainerStarted","Data":"b5b0852b0e6d288efd091375d915014d6b81df6186e96f68a813e23f4e1de09a"}
Sep 18 13:28:59.375794 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:59.375774    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d" event={"ID":"01b21c3d-49a1-4fb5-a925-71ee3ff2904f","Type":"ContainerStarted","Data":"1e63a4dd7825c0d8d0617f0cb43af069be93a4f336757f4bf5cae79cd7a7e151"}
Sep 18 13:28:59.375879 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:59.375812    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d" event={"ID":"01b21c3d-49a1-4fb5-a925-71ee3ff2904f","Type":"ContainerStarted","Data":"6e2dfe88b126b7d4c65a40c9198168341fadc88bd1d35d7d700caec6c7cbb003"}
Sep 18 13:28:59.375879 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:59.375834    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d"
Sep 18 13:28:59.386931 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:59.386877    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2" podStartSLOduration=1.965990503 podStartE2EDuration="5.386863268s" totalImagesPullingTime="3.420872765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:28:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:28:59.385768311 +0000 UTC m=+836.741228149" watchObservedRunningTime="2026-09-18 13:28:59.386863268 +0000 UTC m=+836.742323093"
Sep 18 13:28:59.413542 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:59.413492    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7478-5c0a6c" podStartSLOduration=1.894299927 podStartE2EDuration="5.413473003s" totalImagesPullingTime="3.519173076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:28:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:28:59.410533538 +0000 UTC m=+836.765993364" watchObservedRunningTime="2026-09-18 13:28:59.413473003 +0000 UTC m=+836.768932831"
Sep 18 13:28:59.449825 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:59.449773    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng" podStartSLOduration=2.458652904 podStartE2EDuration="4.44975382s" totalImagesPullingTime="1.991100916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:28:59.44709353 +0000 UTC m=+836.802553353" watchObservedRunningTime="2026-09-18 13:28:59.44975382 +0000 UTC m=+836.805213646"
Sep 18 13:28:59.493129 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:59.492891    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw" podStartSLOduration=2.489907569 podStartE2EDuration="4.492872909s" totalImagesPullingTime="2.00296534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:28:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:28:59.486566023 +0000 UTC m=+836.842025860" watchObservedRunningTime="2026-09-18 13:28:59.492872909 +0000 UTC m=+836.848332735"
Sep 18 13:28:59.537274 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:28:59.537222    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d" podStartSLOduration=3.537206056 podStartE2EDuration="3.537206056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:28:59.535851038 +0000 UTC m=+836.891310863" watchObservedRunningTime="2026-09-18 13:28:59.537206056 +0000 UTC m=+836.892665887"
Sep 18 13:29:00.112768 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:00.112674    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7478-5c0a6c"]
Sep 18 13:29:00.115742 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:00.115713    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7478-5c0a6c"]
Sep 18 13:29:00.380310 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:29:00.380215    2586 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7478-5c0a6c" container="test-curl-7478-5c0a6c"
Sep 18 13:29:01.386007 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:01.385942    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7478-5c0a6c" podUID="b1e5d759-6921-42bf-8150-d54b82f3a2ba" containerName="test-curl-7478-5c0a6c" containerID="cri-o://b5b0852b0e6d288efd091375d915014d6b81df6186e96f68a813e23f4e1de09a" gracePeriod=2
Sep 18 13:29:03.558770 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:03.558736    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-5c0a6c"
Sep 18 13:29:03.609668 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:03.609633    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bl76j\" (UniqueName: \"kubernetes.io/projected/b1e5d759-6921-42bf-8150-d54b82f3a2ba-kube-api-access-bl76j\") pod \"b1e5d759-6921-42bf-8150-d54b82f3a2ba\" (UID: \"b1e5d759-6921-42bf-8150-d54b82f3a2ba\") "
Sep 18 13:29:03.612082 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:03.612061    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1e5d759-6921-42bf-8150-d54b82f3a2ba-kube-api-access-bl76j" (OuterVolumeSpecName: "kube-api-access-bl76j") pod "b1e5d759-6921-42bf-8150-d54b82f3a2ba" (UID: "b1e5d759-6921-42bf-8150-d54b82f3a2ba"). InnerVolumeSpecName "kube-api-access-bl76j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:29:03.710501 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:03.710426    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl76j\" (UniqueName: \"kubernetes.io/projected/b1e5d759-6921-42bf-8150-d54b82f3a2ba-kube-api-access-bl76j\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:29:04.401132 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:04.401097    2586 generic.go:356] "Generic (PLEG): container finished" podID="b1e5d759-6921-42bf-8150-d54b82f3a2ba" containerID="b5b0852b0e6d288efd091375d915014d6b81df6186e96f68a813e23f4e1de09a" exitCode=137
Sep 18 13:29:04.401309 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:04.401154    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-5c0a6c"
Sep 18 13:29:04.401309 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:04.401196    2586 scope.go:117] "RemoveContainer" containerID="b5b0852b0e6d288efd091375d915014d6b81df6186e96f68a813e23f4e1de09a"
Sep 18 13:29:04.410713 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:04.410600    2586 scope.go:117] "RemoveContainer" containerID="b5b0852b0e6d288efd091375d915014d6b81df6186e96f68a813e23f4e1de09a"
Sep 18 13:29:04.410955 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:29:04.410890    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5b0852b0e6d288efd091375d915014d6b81df6186e96f68a813e23f4e1de09a\": container with ID starting with b5b0852b0e6d288efd091375d915014d6b81df6186e96f68a813e23f4e1de09a not found: ID does not exist" containerID="b5b0852b0e6d288efd091375d915014d6b81df6186e96f68a813e23f4e1de09a"
Sep 18 13:29:04.410955 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:04.410941    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b0852b0e6d288efd091375d915014d6b81df6186e96f68a813e23f4e1de09a"} err="failed to get container status \"b5b0852b0e6d288efd091375d915014d6b81df6186e96f68a813e23f4e1de09a\": rpc error: code = NotFound desc = could not find container \"b5b0852b0e6d288efd091375d915014d6b81df6186e96f68a813e23f4e1de09a\": container with ID starting with b5b0852b0e6d288efd091375d915014d6b81df6186e96f68a813e23f4e1de09a not found: ID does not exist"
Sep 18 13:29:04.414158 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:29:04.414133    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7478-5c0a6c\" is forbidden: User \"system:node:ip-10-0-131-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-221.ec2.internal' and this object" podUID="b1e5d759-6921-42bf-8150-d54b82f3a2ba" pod="opendatahub/test-curl-7478-5c0a6c"
Sep 18 13:29:04.850012 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:29:04.849971    2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7478-5c0a6c\" is forbidden: User \"system:node:ip-10-0-131-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-221.ec2.internal' and this object" podUID="b1e5d759-6921-42bf-8150-d54b82f3a2ba" pod="opendatahub/test-curl-7478-5c0a6c"
Sep 18 13:29:04.854362 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:04.854334    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1e5d759-6921-42bf-8150-d54b82f3a2ba" path="/var/lib/kubelet/pods/b1e5d759-6921-42bf-8150-d54b82f3a2ba/volumes"
Sep 18 13:29:06.567372 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:06.567335    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t"]
Sep 18 13:29:06.567718 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:06.567681    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1e5d759-6921-42bf-8150-d54b82f3a2ba" containerName="test-curl-7478-5c0a6c"
Sep 18 13:29:06.567718 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:06.567692    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1e5d759-6921-42bf-8150-d54b82f3a2ba" containerName="test-curl-7478-5c0a6c"
Sep 18 13:29:06.567798 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:06.567771    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1e5d759-6921-42bf-8150-d54b82f3a2ba" containerName="test-curl-7478-5c0a6c"
Sep 18 13:29:06.572250 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:06.572231    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t"
Sep 18 13:29:06.574227 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:06.574207    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a304aa20\""
Sep 18 13:29:06.583458 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:06.583323    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t"]
Sep 18 13:29:06.585872 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:06.585849    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx"]
Sep 18 13:29:06.589258 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:06.589238    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx"
Sep 18 13:29:06.592717 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:06.592688    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a304aa20-dockercfg-qgdng\""
Sep 18 13:29:06.601623 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:06.601584    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx"]
Sep 18 13:29:06.634454 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:06.634420    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n6xm\" (UniqueName: \"kubernetes.io/projected/9f0eaba0-8548-49bd-968f-bc0aa541ab1e-kube-api-access-9n6xm\") pod \"payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t\" (UID: \"9f0eaba0-8548-49bd-968f-bc0aa541ab1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t"
Sep 18 13:29:06.634634 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:06.634474    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8741d72d-1d8d-4e9d-b301-d6a5c6e490e8-plugins-config-volume\") pod \"payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx\" (UID: \"8741d72d-1d8d-4e9d-b301-d6a5c6e490e8\") " pod="openshift-ingress/payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx"
Sep 18 13:29:06.634634 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:06.634493    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nvlp\" (UniqueName: \"kubernetes.io/projected/8741d72d-1d8d-4e9d-b301-d6a5c6e490e8-kube-api-access-7nvlp\") pod \"payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx\" (UID: \"8741d72d-1d8d-4e9d-b301-d6a5c6e490e8\") " pod="openshift-ingress/payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx"
Sep 18 13:29:06.634634 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:06.634522    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0eaba0-8548-49bd-968f-bc0aa541ab1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t\" (UID: \"9f0eaba0-8548-49bd-968f-bc0aa541ab1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t"
Sep 18 13:29:06.736092 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:06.736058    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8741d72d-1d8d-4e9d-b301-d6a5c6e490e8-plugins-config-volume\") pod \"payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx\" (UID: \"8741d72d-1d8d-4e9d-b301-d6a5c6e490e8\") " pod="openshift-ingress/payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx"
Sep 18 13:29:06.736092 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:06.736093    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nvlp\" (UniqueName: \"kubernetes.io/projected/8741d72d-1d8d-4e9d-b301-d6a5c6e490e8-kube-api-access-7nvlp\") pod \"payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx\" (UID: \"8741d72d-1d8d-4e9d-b301-d6a5c6e490e8\") " pod="openshift-ingress/payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx"
Sep 18 13:29:06.736297 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:06.736129    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0eaba0-8548-49bd-968f-bc0aa541ab1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t\" (UID: \"9f0eaba0-8548-49bd-968f-bc0aa541ab1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t"
Sep 18 13:29:06.736297 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:06.736214    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n6xm\" (UniqueName: \"kubernetes.io/projected/9f0eaba0-8548-49bd-968f-bc0aa541ab1e-kube-api-access-9n6xm\") pod \"payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t\" (UID: \"9f0eaba0-8548-49bd-968f-bc0aa541ab1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t"
Sep 18 13:29:06.736754 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:06.736736    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8741d72d-1d8d-4e9d-b301-d6a5c6e490e8-plugins-config-volume\") pod \"payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx\" (UID: \"8741d72d-1d8d-4e9d-b301-d6a5c6e490e8\") " pod="openshift-ingress/payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx"
Sep 18 13:29:06.736806 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:06.736787    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0eaba0-8548-49bd-968f-bc0aa541ab1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t\" (UID: \"9f0eaba0-8548-49bd-968f-bc0aa541ab1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t"
Sep 18 13:29:06.744824 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:06.744797    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n6xm\" (UniqueName: \"kubernetes.io/projected/9f0eaba0-8548-49bd-968f-bc0aa541ab1e-kube-api-access-9n6xm\") pod \"payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t\" (UID: \"9f0eaba0-8548-49bd-968f-bc0aa541ab1e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t"
Sep 18 13:29:06.744976 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:06.744961    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nvlp\" (UniqueName: \"kubernetes.io/projected/8741d72d-1d8d-4e9d-b301-d6a5c6e490e8-kube-api-access-7nvlp\") pod \"payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx\" (UID: \"8741d72d-1d8d-4e9d-b301-d6a5c6e490e8\") " pod="openshift-ingress/payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx"
Sep 18 13:29:06.882437 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:06.882345    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t"
Sep 18 13:29:06.900579 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:06.900553    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx"
Sep 18 13:29:07.171096 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:07.171062    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t"]
Sep 18 13:29:07.174719 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:29:07.174692    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0eaba0_8548_49bd_968f_bc0aa541ab1e.slice/crio-47c9f551b74e19b9edc649213aa570e7cc851ca43bdc52fc358d6c6f8ce3543e WatchSource:0}: Error finding container 47c9f551b74e19b9edc649213aa570e7cc851ca43bdc52fc358d6c6f8ce3543e: Status 404 returned error can't find the container with id 47c9f551b74e19b9edc649213aa570e7cc851ca43bdc52fc358d6c6f8ce3543e
Sep 18 13:29:07.193532 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:07.193508    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx"]
Sep 18 13:29:07.195997 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:29:07.195966    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8741d72d_1d8d_4e9d_b301_d6a5c6e490e8.slice/crio-eaa1aaebd6b746b7945cb94616eafe9dbc19d901d732883bfcaaf8c1c3853532 WatchSource:0}: Error finding container eaa1aaebd6b746b7945cb94616eafe9dbc19d901d732883bfcaaf8c1c3853532: Status 404 returned error can't find the container with id eaa1aaebd6b746b7945cb94616eafe9dbc19d901d732883bfcaaf8c1c3853532
Sep 18 13:29:07.422056 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:07.421953    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx" event={"ID":"8741d72d-1d8d-4e9d-b301-d6a5c6e490e8","Type":"ContainerStarted","Data":"803880731b982d69751baf13c2499afdd8615d41fdcdae49561091b02e097a8e"}
Sep 18 13:29:07.422056 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:07.422015    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx" event={"ID":"8741d72d-1d8d-4e9d-b301-d6a5c6e490e8","Type":"ContainerStarted","Data":"eaa1aaebd6b746b7945cb94616eafe9dbc19d901d732883bfcaaf8c1c3853532"}
Sep 18 13:29:07.422286 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:07.422176    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx"
Sep 18 13:29:07.423685 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:07.423659    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t" event={"ID":"9f0eaba0-8548-49bd-968f-bc0aa541ab1e","Type":"ContainerStarted","Data":"46cecda4f837c085e3ef02d5391a0dfee4376be58b63b9ee2ad48e120879ec32"}
Sep 18 13:29:07.423798 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:07.423704    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t" event={"ID":"9f0eaba0-8548-49bd-968f-bc0aa541ab1e","Type":"ContainerStarted","Data":"47c9f551b74e19b9edc649213aa570e7cc851ca43bdc52fc358d6c6f8ce3543e"}
Sep 18 13:29:07.423798 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:07.423722    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t"
Sep 18 13:29:07.441074 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:07.441028    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx" podStartSLOduration=1.441008403 podStartE2EDuration="1.441008403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:29:07.439807127 +0000 UTC m=+844.795266950" watchObservedRunningTime="2026-09-18 13:29:07.441008403 +0000 UTC m=+844.796468233"
Sep 18 13:29:07.946629 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:07.946571    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t" podStartSLOduration=1.946547 podStartE2EDuration="1.946547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:29:07.457612456 +0000 UTC m=+844.813072280" watchObservedRunningTime="2026-09-18 13:29:07.946547 +0000 UTC m=+845.302006825"
Sep 18 13:29:07.947564 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:07.947542    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:29:10.367595 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.367562    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt"]
Sep 18 13:29:10.371242 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.371221    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt"
Sep 18 13:29:10.374917 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.374879    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-b7292e-dockercfg-2nbw7\""
Sep 18 13:29:10.375291 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.375263    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-b7292e\""
Sep 18 13:29:10.382121 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.382099    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d"
Sep 18 13:29:10.382220 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.382172    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw"
Sep 18 13:29:10.382280 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.382221    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2"
Sep 18 13:29:10.382280 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.382253    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng"
Sep 18 13:29:10.395190 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.395164    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt"]
Sep 18 13:29:10.456615 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.456581    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc"]
Sep 18 13:29:10.460000 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.459984    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc"
Sep 18 13:29:10.470654 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.470629    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xf5l\" (UniqueName: \"kubernetes.io/projected/407c62f1-1a0c-420b-89e9-f379905eebe0-kube-api-access-6xf5l\") pod \"payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt\" (UID: \"407c62f1-1a0c-420b-89e9-f379905eebe0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt"
Sep 18 13:29:10.470829 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.470799    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407c62f1-1a0c-420b-89e9-f379905eebe0-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt\" (UID: \"407c62f1-1a0c-420b-89e9-f379905eebe0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt"
Sep 18 13:29:10.480322 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.480299    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc"]
Sep 18 13:29:10.571985 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.571942    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xf5l\" (UniqueName: \"kubernetes.io/projected/407c62f1-1a0c-420b-89e9-f379905eebe0-kube-api-access-6xf5l\") pod \"payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt\" (UID: \"407c62f1-1a0c-420b-89e9-f379905eebe0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt"
Sep 18 13:29:10.572149 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.572070    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1ec06bc-7c08-42f5-b5b1-493f91be1d15-plugins-config-volume\") pod \"payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc\" (UID: \"d1ec06bc-7c08-42f5-b5b1-493f91be1d15\") " pod="openshift-ingress/payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc"
Sep 18 13:29:10.572149 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.572109    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407c62f1-1a0c-420b-89e9-f379905eebe0-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt\" (UID: \"407c62f1-1a0c-420b-89e9-f379905eebe0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt"
Sep 18 13:29:10.572293 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.572165    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75j52\" (UniqueName: \"kubernetes.io/projected/d1ec06bc-7c08-42f5-b5b1-493f91be1d15-kube-api-access-75j52\") pod \"payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc\" (UID: \"d1ec06bc-7c08-42f5-b5b1-493f91be1d15\") " pod="openshift-ingress/payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc"
Sep 18 13:29:10.573104 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.573079    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407c62f1-1a0c-420b-89e9-f379905eebe0-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt\" (UID: \"407c62f1-1a0c-420b-89e9-f379905eebe0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt"
Sep 18 13:29:10.582979 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.582955    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xf5l\" (UniqueName: \"kubernetes.io/projected/407c62f1-1a0c-420b-89e9-f379905eebe0-kube-api-access-6xf5l\") pod \"payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt\" (UID: \"407c62f1-1a0c-420b-89e9-f379905eebe0\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt"
Sep 18 13:29:10.672798 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.672760    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75j52\" (UniqueName: \"kubernetes.io/projected/d1ec06bc-7c08-42f5-b5b1-493f91be1d15-kube-api-access-75j52\") pod \"payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc\" (UID: \"d1ec06bc-7c08-42f5-b5b1-493f91be1d15\") " pod="openshift-ingress/payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc"
Sep 18 13:29:10.672987 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.672867    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1ec06bc-7c08-42f5-b5b1-493f91be1d15-plugins-config-volume\") pod \"payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc\" (UID: \"d1ec06bc-7c08-42f5-b5b1-493f91be1d15\") " pod="openshift-ingress/payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc"
Sep 18 13:29:10.673563 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.673542    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1ec06bc-7c08-42f5-b5b1-493f91be1d15-plugins-config-volume\") pod \"payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc\" (UID: \"d1ec06bc-7c08-42f5-b5b1-493f91be1d15\") " pod="openshift-ingress/payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc"
Sep 18 13:29:10.682362 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.682330    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt"
Sep 18 13:29:10.682689 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.682669    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75j52\" (UniqueName: \"kubernetes.io/projected/d1ec06bc-7c08-42f5-b5b1-493f91be1d15-kube-api-access-75j52\") pod \"payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc\" (UID: \"d1ec06bc-7c08-42f5-b5b1-493f91be1d15\") " pod="openshift-ingress/payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc"
Sep 18 13:29:10.770293 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.770258    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc"
Sep 18 13:29:10.968875 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:10.968801    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt"]
Sep 18 13:29:10.972347 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:29:10.972323    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod407c62f1_1a0c_420b_89e9_f379905eebe0.slice/crio-f8eb0d10c7f26aac25bdc122a954da524be8a234e4baa9189c5088b7cc85f054 WatchSource:0}: Error finding container f8eb0d10c7f26aac25bdc122a954da524be8a234e4baa9189c5088b7cc85f054: Status 404 returned error can't find the container with id f8eb0d10c7f26aac25bdc122a954da524be8a234e4baa9189c5088b7cc85f054
Sep 18 13:29:11.058642 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:11.058610    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc"]
Sep 18 13:29:11.061463 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:29:11.061434    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ec06bc_7c08_42f5_b5b1_493f91be1d15.slice/crio-13de007d6669d4cd114ecbc2aa846a14062e2dcbfeebe365dc68eb838cb4d309 WatchSource:0}: Error finding container 13de007d6669d4cd114ecbc2aa846a14062e2dcbfeebe365dc68eb838cb4d309: Status 404 returned error can't find the container with id 13de007d6669d4cd114ecbc2aa846a14062e2dcbfeebe365dc68eb838cb4d309
Sep 18 13:29:11.443614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:11.443572    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt" event={"ID":"407c62f1-1a0c-420b-89e9-f379905eebe0","Type":"ContainerStarted","Data":"a3dac07a64b6481a509b760509399df75437b3400c6bab92fb55094b35ea02ce"}
Sep 18 13:29:11.444068 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:11.443640    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt" event={"ID":"407c62f1-1a0c-420b-89e9-f379905eebe0","Type":"ContainerStarted","Data":"f8eb0d10c7f26aac25bdc122a954da524be8a234e4baa9189c5088b7cc85f054"}
Sep 18 13:29:11.444068 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:11.443759    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt"
Sep 18 13:29:11.445238 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:11.445203    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc" event={"ID":"d1ec06bc-7c08-42f5-b5b1-493f91be1d15","Type":"ContainerStarted","Data":"2161bf9ed96070b1f5e2ccfda92f3a28308aa18728c309aef5b09bc135aaeb7c"}
Sep 18 13:29:11.445339 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:11.445251    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc" event={"ID":"d1ec06bc-7c08-42f5-b5b1-493f91be1d15","Type":"ContainerStarted","Data":"13de007d6669d4cd114ecbc2aa846a14062e2dcbfeebe365dc68eb838cb4d309"}
Sep 18 13:29:11.445408 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:11.445339    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc"
Sep 18 13:29:11.560173 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:11.560111    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt" podStartSLOduration=1.5600903270000002 podStartE2EDuration="1.560090327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:29:11.507444479 +0000 UTC m=+848.862904305" watchObservedRunningTime="2026-09-18 13:29:11.560090327 +0000 UTC m=+848.915550154"
Sep 18 13:29:11.561139 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:11.561108    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc" podStartSLOduration=1.561095834 podStartE2EDuration="1.561095834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:29:11.55867214 +0000 UTC m=+848.914131966" watchObservedRunningTime="2026-09-18 13:29:11.561095834 +0000 UTC m=+848.916555673"
Sep 18 13:29:18.353583 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:18.353550    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f"]
Sep 18 13:29:18.356785 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:18.356766    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f"
Sep 18 13:29:18.359351 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:18.359322    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-021eb8\""
Sep 18 13:29:18.359459 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:18.359370    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-021eb8-dockercfg-dhvrf\""
Sep 18 13:29:18.367354 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:18.367332    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f"]
Sep 18 13:29:18.398608 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:18.398578    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn"]
Sep 18 13:29:18.401961 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:18.401943    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn"
Sep 18 13:29:18.412243 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:18.412216    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn"]
Sep 18 13:29:18.431105 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:18.431060    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t"
Sep 18 13:29:18.431278 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:18.431152    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx"
Sep 18 13:29:18.444293 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:18.444261    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce7f3db6-ec3d-4817-89de-ce9aa8a62d24-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f\" (UID: \"ce7f3db6-ec3d-4817-89de-ce9aa8a62d24\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f"
Sep 18 13:29:18.444420 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:18.444308    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkjhr\" (UniqueName: \"kubernetes.io/projected/ce7f3db6-ec3d-4817-89de-ce9aa8a62d24-kube-api-access-jkjhr\") pod \"payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f\" (UID: \"ce7f3db6-ec3d-4817-89de-ce9aa8a62d24\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f"
Sep 18 13:29:18.544962 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:18.544923    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd3d2aa6-f631-4a99-a3e7-be8ef628d1af-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn\" (UID: \"fd3d2aa6-f631-4a99-a3e7-be8ef628d1af\") " pod="openshift-ingress/payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn"
Sep 18 13:29:18.545157 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:18.545012    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce7f3db6-ec3d-4817-89de-ce9aa8a62d24-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f\" (UID: \"ce7f3db6-ec3d-4817-89de-ce9aa8a62d24\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f"
Sep 18 13:29:18.545157 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:18.545094    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkjhr\" (UniqueName: \"kubernetes.io/projected/ce7f3db6-ec3d-4817-89de-ce9aa8a62d24-kube-api-access-jkjhr\") pod \"payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f\" (UID: \"ce7f3db6-ec3d-4817-89de-ce9aa8a62d24\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f"
Sep 18 13:29:18.545157 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:18.545127    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxkm8\" (UniqueName: \"kubernetes.io/projected/fd3d2aa6-f631-4a99-a3e7-be8ef628d1af-kube-api-access-fxkm8\") pod \"payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn\" (UID: \"fd3d2aa6-f631-4a99-a3e7-be8ef628d1af\") " pod="openshift-ingress/payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn"
Sep 18 13:29:18.545827 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:18.545805    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce7f3db6-ec3d-4817-89de-ce9aa8a62d24-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f\" (UID: \"ce7f3db6-ec3d-4817-89de-ce9aa8a62d24\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f"
Sep 18 13:29:18.582500 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:18.582469    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkjhr\" (UniqueName: \"kubernetes.io/projected/ce7f3db6-ec3d-4817-89de-ce9aa8a62d24-kube-api-access-jkjhr\") pod \"payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f\" (UID: \"ce7f3db6-ec3d-4817-89de-ce9aa8a62d24\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f"
Sep 18 13:29:18.646133 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:18.646099    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxkm8\" (UniqueName: \"kubernetes.io/projected/fd3d2aa6-f631-4a99-a3e7-be8ef628d1af-kube-api-access-fxkm8\") pod \"payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn\" (UID: \"fd3d2aa6-f631-4a99-a3e7-be8ef628d1af\") " pod="openshift-ingress/payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn"
Sep 18 13:29:18.646297 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:18.646197    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd3d2aa6-f631-4a99-a3e7-be8ef628d1af-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn\" (UID: \"fd3d2aa6-f631-4a99-a3e7-be8ef628d1af\") " pod="openshift-ingress/payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn"
Sep 18 13:29:18.646895 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:18.646877    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd3d2aa6-f631-4a99-a3e7-be8ef628d1af-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn\" (UID: \"fd3d2aa6-f631-4a99-a3e7-be8ef628d1af\") " pod="openshift-ingress/payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn"
Sep 18 13:29:18.662575 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:18.662547    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxkm8\" (UniqueName: \"kubernetes.io/projected/fd3d2aa6-f631-4a99-a3e7-be8ef628d1af-kube-api-access-fxkm8\") pod \"payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn\" (UID: \"fd3d2aa6-f631-4a99-a3e7-be8ef628d1af\") " pod="openshift-ingress/payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn"
Sep 18 13:29:18.668324 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:18.668293    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f"
Sep 18 13:29:18.715228 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:18.715158    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn"
Sep 18 13:29:18.959765 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:18.959682    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f"]
Sep 18 13:29:18.964400 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:29:18.964372    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7f3db6_ec3d_4817_89de_ce9aa8a62d24.slice/crio-4a9c397b1cc572d07c2ea83063cc12d98841bf15c3766d3c81699969be7e90b7 WatchSource:0}: Error finding container 4a9c397b1cc572d07c2ea83063cc12d98841bf15c3766d3c81699969be7e90b7: Status 404 returned error can't find the container with id 4a9c397b1cc572d07c2ea83063cc12d98841bf15c3766d3c81699969be7e90b7
Sep 18 13:29:19.031008 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:19.030982    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn"]
Sep 18 13:29:19.033034 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:29:19.033010    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3d2aa6_f631_4a99_a3e7_be8ef628d1af.slice/crio-cbfa5c1f8a497265b76fd6086974d87ae436600a722c74c99afb3de6adc24abf WatchSource:0}: Error finding container cbfa5c1f8a497265b76fd6086974d87ae436600a722c74c99afb3de6adc24abf: Status 404 returned error can't find the container with id cbfa5c1f8a497265b76fd6086974d87ae436600a722c74c99afb3de6adc24abf
Sep 18 13:29:19.488520 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:19.488484    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f" event={"ID":"ce7f3db6-ec3d-4817-89de-ce9aa8a62d24","Type":"ContainerStarted","Data":"9fec4614a9a04dcb5027795e3af0f8b5ca09acb132e44e9f1d0d08beb4b1afd4"}
Sep 18 13:29:19.488975 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:19.488539    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f" event={"ID":"ce7f3db6-ec3d-4817-89de-ce9aa8a62d24","Type":"ContainerStarted","Data":"4a9c397b1cc572d07c2ea83063cc12d98841bf15c3766d3c81699969be7e90b7"}
Sep 18 13:29:19.488975 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:19.488567    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f"
Sep 18 13:29:19.490212 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:19.490188    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn" event={"ID":"fd3d2aa6-f631-4a99-a3e7-be8ef628d1af","Type":"ContainerStarted","Data":"8f30e62871ffdd6bfbf09df65457bb99d54885392d737a74dfd98ab5a9410fc7"}
Sep 18 13:29:19.490320 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:19.490234    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn" event={"ID":"fd3d2aa6-f631-4a99-a3e7-be8ef628d1af","Type":"ContainerStarted","Data":"cbfa5c1f8a497265b76fd6086974d87ae436600a722c74c99afb3de6adc24abf"}
Sep 18 13:29:19.490320 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:19.490261    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn"
Sep 18 13:29:19.531271 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:19.531183    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f" podStartSLOduration=1.531164572 podStartE2EDuration="1.531164572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:29:19.530731179 +0000 UTC m=+856.886191004" watchObservedRunningTime="2026-09-18 13:29:19.531164572 +0000 UTC m=+856.886624401"
Sep 18 13:29:22.451768 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:22.451734    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt"
Sep 18 13:29:22.452156 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:22.451834    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc"
Sep 18 13:29:22.536254 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:22.536200    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn" podStartSLOduration=4.536182309 podStartE2EDuration="4.536182309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:29:19.571292328 +0000 UTC m=+856.926752153" watchObservedRunningTime="2026-09-18 13:29:22.536182309 +0000 UTC m=+859.891642143"
Sep 18 13:29:26.366623 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:26.366588    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:29:30.497897 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:30.497863    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn"
Sep 18 13:29:30.498359 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:30.497976    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f"
Sep 18 13:29:31.508575 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:31.508397    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx"]
Sep 18 13:29:31.509104 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:31.508645    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx" podUID="8741d72d-1d8d-4e9d-b301-d6a5c6e490e8" containerName="payload-processing" containerID="cri-o://803880731b982d69751baf13c2499afdd8615d41fdcdae49561091b02e097a8e" gracePeriod=30
Sep 18 13:29:31.512111 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:31.512084    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t"]
Sep 18 13:29:31.512379 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:31.512316    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t" podUID="9f0eaba0-8548-49bd-968f-bc0aa541ab1e" containerName="payload-pre-processing" containerID="cri-o://46cecda4f837c085e3ef02d5391a0dfee4376be58b63b9ee2ad48e120879ec32" gracePeriod=30
Sep 18 13:29:31.791665 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:31.791643    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx"
Sep 18 13:29:31.794193 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:31.794174    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t"
Sep 18 13:29:31.972850 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:31.972818    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nvlp\" (UniqueName: \"kubernetes.io/projected/8741d72d-1d8d-4e9d-b301-d6a5c6e490e8-kube-api-access-7nvlp\") pod \"8741d72d-1d8d-4e9d-b301-d6a5c6e490e8\" (UID: \"8741d72d-1d8d-4e9d-b301-d6a5c6e490e8\") "
Sep 18 13:29:31.973049 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:31.972873    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0eaba0-8548-49bd-968f-bc0aa541ab1e-plugins-config-volume\") pod \"9f0eaba0-8548-49bd-968f-bc0aa541ab1e\" (UID: \"9f0eaba0-8548-49bd-968f-bc0aa541ab1e\") "
Sep 18 13:29:31.973049 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:31.972924    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9n6xm\" (UniqueName: \"kubernetes.io/projected/9f0eaba0-8548-49bd-968f-bc0aa541ab1e-kube-api-access-9n6xm\") pod \"9f0eaba0-8548-49bd-968f-bc0aa541ab1e\" (UID: \"9f0eaba0-8548-49bd-968f-bc0aa541ab1e\") "
Sep 18 13:29:31.973049 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:31.972983    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8741d72d-1d8d-4e9d-b301-d6a5c6e490e8-plugins-config-volume\") pod \"8741d72d-1d8d-4e9d-b301-d6a5c6e490e8\" (UID: \"8741d72d-1d8d-4e9d-b301-d6a5c6e490e8\") "
Sep 18 13:29:31.973285 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:31.973252    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0eaba0-8548-49bd-968f-bc0aa541ab1e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9f0eaba0-8548-49bd-968f-bc0aa541ab1e" (UID: "9f0eaba0-8548-49bd-968f-bc0aa541ab1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:29:31.973530 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:31.973469    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f0eaba0-8548-49bd-968f-bc0aa541ab1e-plugins-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:29:31.973773 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:31.973566    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8741d72d-1d8d-4e9d-b301-d6a5c6e490e8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8741d72d-1d8d-4e9d-b301-d6a5c6e490e8" (UID: "8741d72d-1d8d-4e9d-b301-d6a5c6e490e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:29:31.975869 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:31.975838    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8741d72d-1d8d-4e9d-b301-d6a5c6e490e8-kube-api-access-7nvlp" (OuterVolumeSpecName: "kube-api-access-7nvlp") pod "8741d72d-1d8d-4e9d-b301-d6a5c6e490e8" (UID: "8741d72d-1d8d-4e9d-b301-d6a5c6e490e8"). InnerVolumeSpecName "kube-api-access-7nvlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:29:31.976002 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:31.975972    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0eaba0-8548-49bd-968f-bc0aa541ab1e-kube-api-access-9n6xm" (OuterVolumeSpecName: "kube-api-access-9n6xm") pod "9f0eaba0-8548-49bd-968f-bc0aa541ab1e" (UID: "9f0eaba0-8548-49bd-968f-bc0aa541ab1e"). InnerVolumeSpecName "kube-api-access-9n6xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:29:32.074125 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:32.074052    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nvlp\" (UniqueName: \"kubernetes.io/projected/8741d72d-1d8d-4e9d-b301-d6a5c6e490e8-kube-api-access-7nvlp\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:29:32.074125 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:32.074077    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n6xm\" (UniqueName: \"kubernetes.io/projected/9f0eaba0-8548-49bd-968f-bc0aa541ab1e-kube-api-access-9n6xm\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:29:32.074125 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:32.074087    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8741d72d-1d8d-4e9d-b301-d6a5c6e490e8-plugins-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:29:32.557844 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:32.557813    2586 generic.go:356] "Generic (PLEG): container finished" podID="8741d72d-1d8d-4e9d-b301-d6a5c6e490e8" containerID="803880731b982d69751baf13c2499afdd8615d41fdcdae49561091b02e097a8e" exitCode=0
Sep 18 13:29:32.558332 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:32.557865    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx"
Sep 18 13:29:32.558332 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:32.557895    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx" event={"ID":"8741d72d-1d8d-4e9d-b301-d6a5c6e490e8","Type":"ContainerDied","Data":"803880731b982d69751baf13c2499afdd8615d41fdcdae49561091b02e097a8e"}
Sep 18 13:29:32.558332 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:32.557977    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx" event={"ID":"8741d72d-1d8d-4e9d-b301-d6a5c6e490e8","Type":"ContainerDied","Data":"eaa1aaebd6b746b7945cb94616eafe9dbc19d901d732883bfcaaf8c1c3853532"}
Sep 18 13:29:32.558332 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:32.558005    2586 scope.go:117] "RemoveContainer" containerID="803880731b982d69751baf13c2499afdd8615d41fdcdae49561091b02e097a8e"
Sep 18 13:29:32.559471 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:32.559204    2586 generic.go:356] "Generic (PLEG): container finished" podID="9f0eaba0-8548-49bd-968f-bc0aa541ab1e" containerID="46cecda4f837c085e3ef02d5391a0dfee4376be58b63b9ee2ad48e120879ec32" exitCode=0
Sep 18 13:29:32.559471 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:32.559264    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t"
Sep 18 13:29:32.559471 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:32.559284    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t" event={"ID":"9f0eaba0-8548-49bd-968f-bc0aa541ab1e","Type":"ContainerDied","Data":"46cecda4f837c085e3ef02d5391a0dfee4376be58b63b9ee2ad48e120879ec32"}
Sep 18 13:29:32.559471 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:32.559317    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t" event={"ID":"9f0eaba0-8548-49bd-968f-bc0aa541ab1e","Type":"ContainerDied","Data":"47c9f551b74e19b9edc649213aa570e7cc851ca43bdc52fc358d6c6f8ce3543e"}
Sep 18 13:29:32.568470 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:32.568450    2586 scope.go:117] "RemoveContainer" containerID="803880731b982d69751baf13c2499afdd8615d41fdcdae49561091b02e097a8e"
Sep 18 13:29:32.568734 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:29:32.568715    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"803880731b982d69751baf13c2499afdd8615d41fdcdae49561091b02e097a8e\": container with ID starting with 803880731b982d69751baf13c2499afdd8615d41fdcdae49561091b02e097a8e not found: ID does not exist" containerID="803880731b982d69751baf13c2499afdd8615d41fdcdae49561091b02e097a8e"
Sep 18 13:29:32.568808 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:32.568745    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"803880731b982d69751baf13c2499afdd8615d41fdcdae49561091b02e097a8e"} err="failed to get container status \"803880731b982d69751baf13c2499afdd8615d41fdcdae49561091b02e097a8e\": rpc error: code = NotFound desc = could not find container \"803880731b982d69751baf13c2499afdd8615d41fdcdae49561091b02e097a8e\": container with ID starting with 803880731b982d69751baf13c2499afdd8615d41fdcdae49561091b02e097a8e not found: ID does not exist"
Sep 18 13:29:32.568808 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:32.568767    2586 scope.go:117] "RemoveContainer" containerID="46cecda4f837c085e3ef02d5391a0dfee4376be58b63b9ee2ad48e120879ec32"
Sep 18 13:29:32.577015 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:32.576996    2586 scope.go:117] "RemoveContainer" containerID="46cecda4f837c085e3ef02d5391a0dfee4376be58b63b9ee2ad48e120879ec32"
Sep 18 13:29:32.577273 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:29:32.577252    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46cecda4f837c085e3ef02d5391a0dfee4376be58b63b9ee2ad48e120879ec32\": container with ID starting with 46cecda4f837c085e3ef02d5391a0dfee4376be58b63b9ee2ad48e120879ec32 not found: ID does not exist" containerID="46cecda4f837c085e3ef02d5391a0dfee4376be58b63b9ee2ad48e120879ec32"
Sep 18 13:29:32.577337 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:32.577279    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46cecda4f837c085e3ef02d5391a0dfee4376be58b63b9ee2ad48e120879ec32"} err="failed to get container status \"46cecda4f837c085e3ef02d5391a0dfee4376be58b63b9ee2ad48e120879ec32\": rpc error: code = NotFound desc = could not find container \"46cecda4f837c085e3ef02d5391a0dfee4376be58b63b9ee2ad48e120879ec32\": container with ID starting with 46cecda4f837c085e3ef02d5391a0dfee4376be58b63b9ee2ad48e120879ec32 not found: ID does not exist"
Sep 18 13:29:32.599863 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:32.599836    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx"]
Sep 18 13:29:32.609973 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:32.609800    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a304aa20-7bf9b76fc5-2j6hx"]
Sep 18 13:29:32.629752 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:32.629729    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t"]
Sep 18 13:29:32.645281 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:32.645254    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a304aa20-6cc7bf966b-gqf9t"]
Sep 18 13:29:32.852354 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:32.852276    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8741d72d-1d8d-4e9d-b301-d6a5c6e490e8" path="/var/lib/kubelet/pods/8741d72d-1d8d-4e9d-b301-d6a5c6e490e8/volumes"
Sep 18 13:29:32.852633 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:32.852621    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f0eaba0-8548-49bd-968f-bc0aa541ab1e" path="/var/lib/kubelet/pods/9f0eaba0-8548-49bd-968f-bc0aa541ab1e/volumes"
Sep 18 13:29:40.212443 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:40.212404    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj"]
Sep 18 13:29:40.212854 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:40.212840    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8741d72d-1d8d-4e9d-b301-d6a5c6e490e8" containerName="payload-processing"
Sep 18 13:29:40.212933 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:40.212855    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="8741d72d-1d8d-4e9d-b301-d6a5c6e490e8" containerName="payload-processing"
Sep 18 13:29:40.212933 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:40.212863    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f0eaba0-8548-49bd-968f-bc0aa541ab1e" containerName="payload-pre-processing"
Sep 18 13:29:40.212933 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:40.212869    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f0eaba0-8548-49bd-968f-bc0aa541ab1e" containerName="payload-pre-processing"
Sep 18 13:29:40.213035 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:40.212947    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="8741d72d-1d8d-4e9d-b301-d6a5c6e490e8" containerName="payload-processing"
Sep 18 13:29:40.213035 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:40.212962    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f0eaba0-8548-49bd-968f-bc0aa541ab1e" containerName="payload-pre-processing"
Sep 18 13:29:40.216157 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:40.216137    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj"
Sep 18 13:29:40.218219 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:40.218199    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-8178145c-dockercfg-f2qqg\""
Sep 18 13:29:40.218313 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:40.218200    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-8178145c\""
Sep 18 13:29:40.233018 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:40.232984    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj"]
Sep 18 13:29:40.247875 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:40.247840    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7882f5-d821-49eb-9fd6-58f5d068122a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj\" (UID: \"4e7882f5-d821-49eb-9fd6-58f5d068122a\") " pod="openshift-ingress/payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj"
Sep 18 13:29:40.248034 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:40.247884    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkfkt\" (UniqueName: \"kubernetes.io/projected/4e7882f5-d821-49eb-9fd6-58f5d068122a-kube-api-access-qkfkt\") pod \"payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj\" (UID: \"4e7882f5-d821-49eb-9fd6-58f5d068122a\") " pod="openshift-ingress/payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj"
Sep 18 13:29:40.349163 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:40.349123    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7882f5-d821-49eb-9fd6-58f5d068122a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj\" (UID: \"4e7882f5-d821-49eb-9fd6-58f5d068122a\") " pod="openshift-ingress/payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj"
Sep 18 13:29:40.349334 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:40.349173    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkfkt\" (UniqueName: \"kubernetes.io/projected/4e7882f5-d821-49eb-9fd6-58f5d068122a-kube-api-access-qkfkt\") pod \"payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj\" (UID: \"4e7882f5-d821-49eb-9fd6-58f5d068122a\") " pod="openshift-ingress/payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj"
Sep 18 13:29:40.349821 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:40.349802    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7882f5-d821-49eb-9fd6-58f5d068122a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj\" (UID: \"4e7882f5-d821-49eb-9fd6-58f5d068122a\") " pod="openshift-ingress/payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj"
Sep 18 13:29:40.357494 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:40.357462    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkfkt\" (UniqueName: \"kubernetes.io/projected/4e7882f5-d821-49eb-9fd6-58f5d068122a-kube-api-access-qkfkt\") pod \"payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj\" (UID: \"4e7882f5-d821-49eb-9fd6-58f5d068122a\") " pod="openshift-ingress/payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj"
Sep 18 13:29:40.526746 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:40.526664    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj"
Sep 18 13:29:40.824969 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:40.824945    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj"]
Sep 18 13:29:40.827502 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:29:40.827474    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7882f5_d821_49eb_9fd6_58f5d068122a.slice/crio-a36d22997f3771354d1ef259c4f65bb03eacb050a238d9a1faea4c505ccb19c1 WatchSource:0}: Error finding container a36d22997f3771354d1ef259c4f65bb03eacb050a238d9a1faea4c505ccb19c1: Status 404 returned error can't find the container with id a36d22997f3771354d1ef259c4f65bb03eacb050a238d9a1faea4c505ccb19c1
Sep 18 13:29:41.609561 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:41.609525    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj" event={"ID":"4e7882f5-d821-49eb-9fd6-58f5d068122a","Type":"ContainerStarted","Data":"aafef3766298532fab80b8fadee3e99b3b4080cef4416d89ea272d2e5fed4828"}
Sep 18 13:29:41.609973 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:41.609588    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj" event={"ID":"4e7882f5-d821-49eb-9fd6-58f5d068122a","Type":"ContainerStarted","Data":"a36d22997f3771354d1ef259c4f65bb03eacb050a238d9a1faea4c505ccb19c1"}
Sep 18 13:29:41.609973 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:41.609678    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj"
Sep 18 13:29:41.632884 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:41.632837    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj" podStartSLOduration=1.6328201070000001 podStartE2EDuration="1.632820107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:29:41.6323192 +0000 UTC m=+878.987779026" watchObservedRunningTime="2026-09-18 13:29:41.632820107 +0000 UTC m=+878.988279932"
Sep 18 13:29:44.477229 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:44.477186    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:29:52.617504 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:29:52.617472    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj"
Sep 18 13:30:00.171770 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:00.171734    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-021eb8-29828970-h4css"]
Sep 18 13:30:00.174604 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:00.174572    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-021eb8-29828970-h4css"
Sep 18 13:30:00.176850 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:00.176826    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-951469-29828970-tpw7s"]
Sep 18 13:30:00.178856 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:00.178837    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nh2cd\""
Sep 18 13:30:00.179172 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:00.179157    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-951469-29828970-tpw7s"
Sep 18 13:30:00.186491 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:00.186474    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-021eb8-29828970-h4css"
Sep 18 13:30:00.193144 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:00.193118    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-021eb8-29828970-h4css"]
Sep 18 13:30:00.193300 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:00.193283    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-951469-29828970-tpw7s"
Sep 18 13:30:00.205988 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:00.205335    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-951469-29828970-tpw7s"]
Sep 18 13:30:00.210726 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:00.210304    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b7292e-29828970-qdv8s"]
Sep 18 13:30:00.215855 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:00.215828    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b7292e-29828970-qdv8s"
Sep 18 13:30:00.233382 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:00.233333    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b7292e-29828970-qdv8s"
Sep 18 13:30:00.237952 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:00.237663    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b7292e-29828970-qdv8s"]
Sep 18 13:30:00.485324 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:00.485296    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-021eb8-29828970-h4css"]
Sep 18 13:30:00.492510 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:00.492480    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-951469-29828970-tpw7s"]
Sep 18 13:30:00.494102 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:30:00.494073    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b28228_52f3_4edd_b1af_15438f5ea33b.slice/crio-3f64867943966e90722acad1ec29a78ebc438c0f438fd900a26dceda4a01ca18 WatchSource:0}: Error finding container 3f64867943966e90722acad1ec29a78ebc438c0f438fd900a26dceda4a01ca18: Status 404 returned error can't find the container with id 3f64867943966e90722acad1ec29a78ebc438c0f438fd900a26dceda4a01ca18
Sep 18 13:30:00.532927 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:00.532873    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b7292e-29828970-qdv8s"]
Sep 18 13:30:00.535702 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:30:00.535676    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0cc7a30_26f3_4833_8373_869af079e533.slice/crio-76d96f2a24905039cc3878bc990c08cd556fac58063ac701b155dd14f88ba5cd WatchSource:0}: Error finding container 76d96f2a24905039cc3878bc990c08cd556fac58063ac701b155dd14f88ba5cd: Status 404 returned error can't find the container with id 76d96f2a24905039cc3878bc990c08cd556fac58063ac701b155dd14f88ba5cd
Sep 18 13:30:00.702962 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:00.702891    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b7292e-29828970-qdv8s" event={"ID":"e0cc7a30-26f3-4833-8373-869af079e533","Type":"ContainerStarted","Data":"76d96f2a24905039cc3878bc990c08cd556fac58063ac701b155dd14f88ba5cd"}
Sep 18 13:30:00.704522 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:00.704491    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-021eb8-29828970-h4css" event={"ID":"b120ea14-f6cb-41ce-afe4-33b41cf7a44c","Type":"ContainerStarted","Data":"9bb933b78bdc093fe51cd6bf6f56d77aa8525cf4f380c9b60403c7e96cf281af"}
Sep 18 13:30:00.706038 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:00.706011    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-951469-29828970-tpw7s" event={"ID":"b8b28228-52f3-4edd-b1af-15438f5ea33b","Type":"ContainerStarted","Data":"3f64867943966e90722acad1ec29a78ebc438c0f438fd900a26dceda4a01ca18"}
Sep 18 13:30:01.072106 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:01.072059    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:30:03.726047 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:03.726012    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-021eb8-29828970-h4css" event={"ID":"b120ea14-f6cb-41ce-afe4-33b41cf7a44c","Type":"ContainerStarted","Data":"015da423b77568bcd171be7756313cf2e41f739a60562aa60bfb5c5830e5d332"}
Sep 18 13:30:03.727469 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:03.727440    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-951469-29828970-tpw7s" event={"ID":"b8b28228-52f3-4edd-b1af-15438f5ea33b","Type":"ContainerStarted","Data":"94d11ab2109a0c29479f594c9ecd967b5533bf351e37a70dc8f346995709e40b"}
Sep 18 13:30:03.729116 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:03.729095    2586 generic.go:356] "Generic (PLEG): container finished" podID="e0cc7a30-26f3-4833-8373-869af079e533" containerID="97693b1842c3068d669753a562492323f5ed855bf0dd17a5688e154aa263a982" exitCode=0
Sep 18 13:30:03.729203 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:03.729121    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b7292e-29828970-qdv8s" event={"ID":"e0cc7a30-26f3-4833-8373-869af079e533","Type":"ContainerDied","Data":"97693b1842c3068d669753a562492323f5ed855bf0dd17a5688e154aa263a982"}
Sep 18 13:30:03.806266 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:03.806204    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-021eb8-29828970-h4css" podStartSLOduration=1.654108506 podStartE2EDuration="3.806181455s" totalImagesPullingTime="2.152072949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:30:03.777695983 +0000 UTC m=+901.133155818" watchObservedRunningTime="2026-09-18 13:30:03.806181455 +0000 UTC m=+901.161641282"
Sep 18 13:30:03.855439 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:03.855389    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-951469-29828970-tpw7s" podStartSLOduration=1.697093566 podStartE2EDuration="3.855371933s" totalImagesPullingTime="2.158278367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:30:03.85325795 +0000 UTC m=+901.208717774" watchObservedRunningTime="2026-09-18 13:30:03.855371933 +0000 UTC m=+901.210831757"
Sep 18 13:30:04.740471 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:04.740428    2586 generic.go:356] "Generic (PLEG): container finished" podID="b120ea14-f6cb-41ce-afe4-33b41cf7a44c" containerID="015da423b77568bcd171be7756313cf2e41f739a60562aa60bfb5c5830e5d332" exitCode=0
Sep 18 13:30:04.740931 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:04.740506    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-021eb8-29828970-h4css" event={"ID":"b120ea14-f6cb-41ce-afe4-33b41cf7a44c","Type":"ContainerDied","Data":"015da423b77568bcd171be7756313cf2e41f739a60562aa60bfb5c5830e5d332"}
Sep 18 13:30:04.742151 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:04.742127    2586 generic.go:356] "Generic (PLEG): container finished" podID="b8b28228-52f3-4edd-b1af-15438f5ea33b" containerID="94d11ab2109a0c29479f594c9ecd967b5533bf351e37a70dc8f346995709e40b" exitCode=0
Sep 18 13:30:04.742245 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:04.742204    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-951469-29828970-tpw7s" event={"ID":"b8b28228-52f3-4edd-b1af-15438f5ea33b","Type":"ContainerDied","Data":"94d11ab2109a0c29479f594c9ecd967b5533bf351e37a70dc8f346995709e40b"}
Sep 18 13:30:04.828629 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:04.828599    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lcqcw_1cc32441-1189-4113-bf17-2769fa23a747/ovn-acl-logging/0.log"
Sep 18 13:30:04.829860 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:04.829839    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lcqcw_1cc32441-1189-4113-bf17-2769fa23a747/ovn-acl-logging/0.log"
Sep 18 13:30:04.879727 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:04.879699    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b7292e-29828970-qdv8s"
Sep 18 13:30:05.748179 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:05.748149    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b7292e-29828970-qdv8s"
Sep 18 13:30:05.748179 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:05.748164    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b7292e-29828970-qdv8s" event={"ID":"e0cc7a30-26f3-4833-8373-869af079e533","Type":"ContainerDied","Data":"76d96f2a24905039cc3878bc990c08cd556fac58063ac701b155dd14f88ba5cd"}
Sep 18 13:30:05.748670 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:05.748218    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76d96f2a24905039cc3878bc990c08cd556fac58063ac701b155dd14f88ba5cd"
Sep 18 13:30:05.899619 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:05.899588    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-021eb8-29828970-h4css"
Sep 18 13:30:05.928100 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:05.928072    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-951469-29828970-tpw7s"
Sep 18 13:30:06.754774 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:06.754738    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-021eb8-29828970-h4css" event={"ID":"b120ea14-f6cb-41ce-afe4-33b41cf7a44c","Type":"ContainerDied","Data":"9bb933b78bdc093fe51cd6bf6f56d77aa8525cf4f380c9b60403c7e96cf281af"}
Sep 18 13:30:06.755244 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:06.754819    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bb933b78bdc093fe51cd6bf6f56d77aa8525cf4f380c9b60403c7e96cf281af"
Sep 18 13:30:06.755244 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:06.754750    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-021eb8-29828970-h4css"
Sep 18 13:30:06.756097 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:06.756075    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-951469-29828970-tpw7s" event={"ID":"b8b28228-52f3-4edd-b1af-15438f5ea33b","Type":"ContainerDied","Data":"3f64867943966e90722acad1ec29a78ebc438c0f438fd900a26dceda4a01ca18"}
Sep 18 13:30:06.756097 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:06.756093    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-951469-29828970-tpw7s"
Sep 18 13:30:06.756223 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:06.756111    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f64867943966e90722acad1ec29a78ebc438c0f438fd900a26dceda4a01ca18"
Sep 18 13:30:17.592276 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:17.592238    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj"]
Sep 18 13:30:17.592747 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:17.592467    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj" podUID="4e7882f5-d821-49eb-9fd6-58f5d068122a" containerName="payload-processing" containerID="cri-o://aafef3766298532fab80b8fadee3e99b3b4080cef4416d89ea272d2e5fed4828" gracePeriod=30
Sep 18 13:30:17.814553 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:17.814520    2586 generic.go:356] "Generic (PLEG): container finished" podID="4e7882f5-d821-49eb-9fd6-58f5d068122a" containerID="aafef3766298532fab80b8fadee3e99b3b4080cef4416d89ea272d2e5fed4828" exitCode=0
Sep 18 13:30:17.814553 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:17.814564    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj" event={"ID":"4e7882f5-d821-49eb-9fd6-58f5d068122a","Type":"ContainerDied","Data":"aafef3766298532fab80b8fadee3e99b3b4080cef4416d89ea272d2e5fed4828"}
Sep 18 13:30:17.849069 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:17.849006    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj"
Sep 18 13:30:18.009351 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:18.009317    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7882f5-d821-49eb-9fd6-58f5d068122a-plugins-config-volume\") pod \"4e7882f5-d821-49eb-9fd6-58f5d068122a\" (UID: \"4e7882f5-d821-49eb-9fd6-58f5d068122a\") "
Sep 18 13:30:18.009506 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:18.009374    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkfkt\" (UniqueName: \"kubernetes.io/projected/4e7882f5-d821-49eb-9fd6-58f5d068122a-kube-api-access-qkfkt\") pod \"4e7882f5-d821-49eb-9fd6-58f5d068122a\" (UID: \"4e7882f5-d821-49eb-9fd6-58f5d068122a\") "
Sep 18 13:30:18.009795 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:18.009763    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7882f5-d821-49eb-9fd6-58f5d068122a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4e7882f5-d821-49eb-9fd6-58f5d068122a" (UID: "4e7882f5-d821-49eb-9fd6-58f5d068122a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:30:18.011626 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:18.011607    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7882f5-d821-49eb-9fd6-58f5d068122a-kube-api-access-qkfkt" (OuterVolumeSpecName: "kube-api-access-qkfkt") pod "4e7882f5-d821-49eb-9fd6-58f5d068122a" (UID: "4e7882f5-d821-49eb-9fd6-58f5d068122a"). InnerVolumeSpecName "kube-api-access-qkfkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:30:18.110489 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:18.110411    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkfkt\" (UniqueName: \"kubernetes.io/projected/4e7882f5-d821-49eb-9fd6-58f5d068122a-kube-api-access-qkfkt\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:30:18.110489 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:18.110439    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e7882f5-d821-49eb-9fd6-58f5d068122a-plugins-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:30:18.820870 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:18.820838    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj"
Sep 18 13:30:18.821307 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:18.820845    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj" event={"ID":"4e7882f5-d821-49eb-9fd6-58f5d068122a","Type":"ContainerDied","Data":"a36d22997f3771354d1ef259c4f65bb03eacb050a238d9a1faea4c505ccb19c1"}
Sep 18 13:30:18.821307 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:18.820981    2586 scope.go:117] "RemoveContainer" containerID="aafef3766298532fab80b8fadee3e99b3b4080cef4416d89ea272d2e5fed4828"
Sep 18 13:30:18.866204 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:18.865059    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj"]
Sep 18 13:30:18.869775 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:18.869746    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8178145c-5b95fb5998-jp8zj"]
Sep 18 13:30:20.849634 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:20.849601    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e7882f5-d821-49eb-9fd6-58f5d068122a" path="/var/lib/kubelet/pods/4e7882f5-d821-49eb-9fd6-58f5d068122a/volumes"
Sep 18 13:30:21.280788 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:21.280749    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:30:26.396250 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:26.396214    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j"]
Sep 18 13:30:26.396926 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:26.396884    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e7882f5-d821-49eb-9fd6-58f5d068122a" containerName="payload-processing"
Sep 18 13:30:26.397040 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:26.396928    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e7882f5-d821-49eb-9fd6-58f5d068122a" containerName="payload-processing"
Sep 18 13:30:26.397040 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:26.396948    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b120ea14-f6cb-41ce-afe4-33b41cf7a44c" containerName="cleanup"
Sep 18 13:30:26.397040 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:26.396956    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b120ea14-f6cb-41ce-afe4-33b41cf7a44c" containerName="cleanup"
Sep 18 13:30:26.397040 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:26.396967    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b28228-52f3-4edd-b1af-15438f5ea33b" containerName="cleanup"
Sep 18 13:30:26.397040 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:26.396976    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b28228-52f3-4edd-b1af-15438f5ea33b" containerName="cleanup"
Sep 18 13:30:26.397040 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:26.396994    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0cc7a30-26f3-4833-8373-869af079e533" containerName="cleanup"
Sep 18 13:30:26.397040 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:26.397002    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0cc7a30-26f3-4833-8373-869af079e533" containerName="cleanup"
Sep 18 13:30:26.397376 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:26.397129    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b120ea14-f6cb-41ce-afe4-33b41cf7a44c" containerName="cleanup"
Sep 18 13:30:26.397376 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:26.397145    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b28228-52f3-4edd-b1af-15438f5ea33b" containerName="cleanup"
Sep 18 13:30:26.397376 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:26.397154    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0cc7a30-26f3-4833-8373-869af079e533" containerName="cleanup"
Sep 18 13:30:26.397376 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:26.397162    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e7882f5-d821-49eb-9fd6-58f5d068122a" containerName="payload-processing"
Sep 18 13:30:26.401883 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:26.401860    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j"
Sep 18 13:30:26.441754 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:26.441725    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-92246a6e\""
Sep 18 13:30:26.453379 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:26.450053    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j"]
Sep 18 13:30:26.455853 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:26.455828    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-92246a6e-dockercfg-gvltw\""
Sep 18 13:30:26.482393 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:26.482337    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/750e0078-8a09-40d3-b543-3bcf7fe984b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j\" (UID: \"750e0078-8a09-40d3-b543-3bcf7fe984b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j"
Sep 18 13:30:26.482542 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:26.482410    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msrk6\" (UniqueName: \"kubernetes.io/projected/750e0078-8a09-40d3-b543-3bcf7fe984b1-kube-api-access-msrk6\") pod \"payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j\" (UID: \"750e0078-8a09-40d3-b543-3bcf7fe984b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j"
Sep 18 13:30:26.584046 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:26.584007    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/750e0078-8a09-40d3-b543-3bcf7fe984b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j\" (UID: \"750e0078-8a09-40d3-b543-3bcf7fe984b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j"
Sep 18 13:30:26.584224 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:26.584062    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msrk6\" (UniqueName: \"kubernetes.io/projected/750e0078-8a09-40d3-b543-3bcf7fe984b1-kube-api-access-msrk6\") pod \"payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j\" (UID: \"750e0078-8a09-40d3-b543-3bcf7fe984b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j"
Sep 18 13:30:26.584707 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:26.584687    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/750e0078-8a09-40d3-b543-3bcf7fe984b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j\" (UID: \"750e0078-8a09-40d3-b543-3bcf7fe984b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j"
Sep 18 13:30:26.600602 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:26.600577    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msrk6\" (UniqueName: \"kubernetes.io/projected/750e0078-8a09-40d3-b543-3bcf7fe984b1-kube-api-access-msrk6\") pod \"payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j\" (UID: \"750e0078-8a09-40d3-b543-3bcf7fe984b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j"
Sep 18 13:30:26.716079 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:26.715993    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j"
Sep 18 13:30:27.008560 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:27.008526    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j"]
Sep 18 13:30:27.011687 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:30:27.011646    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750e0078_8a09_40d3_b543_3bcf7fe984b1.slice/crio-48c37afa2760000f087022f6eb37e6250259147266a42c636ca64ea38f32b237 WatchSource:0}: Error finding container 48c37afa2760000f087022f6eb37e6250259147266a42c636ca64ea38f32b237: Status 404 returned error can't find the container with id 48c37afa2760000f087022f6eb37e6250259147266a42c636ca64ea38f32b237
Sep 18 13:30:27.867363 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:27.867324    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j" event={"ID":"750e0078-8a09-40d3-b543-3bcf7fe984b1","Type":"ContainerStarted","Data":"42728103f470d72e99e218dfb3747d805119f0c9c3b1431b273979a411f90ad9"}
Sep 18 13:30:27.867809 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:27.867372    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j" event={"ID":"750e0078-8a09-40d3-b543-3bcf7fe984b1","Type":"ContainerStarted","Data":"48c37afa2760000f087022f6eb37e6250259147266a42c636ca64ea38f32b237"}
Sep 18 13:30:27.867809 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:27.867407    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j"
Sep 18 13:30:38.873790 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:38.873757    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j"
Sep 18 13:30:38.892568 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:38.892518    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j" podStartSLOduration=12.892502532 podStartE2EDuration="12.892502532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:30:27.90524502 +0000 UTC m=+925.260704849" watchObservedRunningTime="2026-09-18 13:30:38.892502532 +0000 UTC m=+936.247962356"
Sep 18 13:30:41.993224 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:41.993190    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:30:51.195383 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.195347    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"]
Sep 18 13:30:51.198313 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.198285    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:30:51.200491 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.200471    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-b7292e\"/\"kube-root-ca.crt\""
Sep 18 13:30:51.201022 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.201004    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-b7292e\"/\"default-dockercfg-qx8pg\""
Sep 18 13:30:51.201144 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.201004    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-b7292e\"/\"openshift-service-ca.crt\""
Sep 18 13:30:51.201144 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.201099    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-b7292e\"/\"autores-model-b7292e-kserve-self-signed-certs\""
Sep 18 13:30:51.215955 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.215924    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"]
Sep 18 13:30:51.306924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.306869    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b74a19be-f798-42d0-88be-dfcc26e526f3-tmp-dir\") pod \"autores-model-b7292e-kserve-76f965587-9gshm\" (UID: \"b74a19be-f798-42d0-88be-dfcc26e526f3\") " pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:30:51.307116 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.306970    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr5rp\" (UniqueName: \"kubernetes.io/projected/b74a19be-f798-42d0-88be-dfcc26e526f3-kube-api-access-wr5rp\") pod \"autores-model-b7292e-kserve-76f965587-9gshm\" (UID: \"b74a19be-f798-42d0-88be-dfcc26e526f3\") " pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:30:51.307116 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.307009    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b74a19be-f798-42d0-88be-dfcc26e526f3-dshm\") pod \"autores-model-b7292e-kserve-76f965587-9gshm\" (UID: \"b74a19be-f798-42d0-88be-dfcc26e526f3\") " pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:30:51.307116 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.307036    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b74a19be-f798-42d0-88be-dfcc26e526f3-home\") pod \"autores-model-b7292e-kserve-76f965587-9gshm\" (UID: \"b74a19be-f798-42d0-88be-dfcc26e526f3\") " pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:30:51.307116 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.307055    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b74a19be-f798-42d0-88be-dfcc26e526f3-model-cache\") pod \"autores-model-b7292e-kserve-76f965587-9gshm\" (UID: \"b74a19be-f798-42d0-88be-dfcc26e526f3\") " pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:30:51.307116 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.307079    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b74a19be-f798-42d0-88be-dfcc26e526f3-tls-certs\") pod \"autores-model-b7292e-kserve-76f965587-9gshm\" (UID: \"b74a19be-f798-42d0-88be-dfcc26e526f3\") " pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:30:51.408428 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.408385    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr5rp\" (UniqueName: \"kubernetes.io/projected/b74a19be-f798-42d0-88be-dfcc26e526f3-kube-api-access-wr5rp\") pod \"autores-model-b7292e-kserve-76f965587-9gshm\" (UID: \"b74a19be-f798-42d0-88be-dfcc26e526f3\") " pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:30:51.408608 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.408456    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b74a19be-f798-42d0-88be-dfcc26e526f3-dshm\") pod \"autores-model-b7292e-kserve-76f965587-9gshm\" (UID: \"b74a19be-f798-42d0-88be-dfcc26e526f3\") " pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:30:51.408608 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.408492    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b74a19be-f798-42d0-88be-dfcc26e526f3-home\") pod \"autores-model-b7292e-kserve-76f965587-9gshm\" (UID: \"b74a19be-f798-42d0-88be-dfcc26e526f3\") " pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:30:51.408608 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.408516    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b74a19be-f798-42d0-88be-dfcc26e526f3-model-cache\") pod \"autores-model-b7292e-kserve-76f965587-9gshm\" (UID: \"b74a19be-f798-42d0-88be-dfcc26e526f3\") " pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:30:51.408608 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.408537    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b74a19be-f798-42d0-88be-dfcc26e526f3-tls-certs\") pod \"autores-model-b7292e-kserve-76f965587-9gshm\" (UID: \"b74a19be-f798-42d0-88be-dfcc26e526f3\") " pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:30:51.408608 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.408608    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b74a19be-f798-42d0-88be-dfcc26e526f3-tmp-dir\") pod \"autores-model-b7292e-kserve-76f965587-9gshm\" (UID: \"b74a19be-f798-42d0-88be-dfcc26e526f3\") " pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:30:51.408983 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.408958    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b74a19be-f798-42d0-88be-dfcc26e526f3-home\") pod \"autores-model-b7292e-kserve-76f965587-9gshm\" (UID: \"b74a19be-f798-42d0-88be-dfcc26e526f3\") " pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:30:51.409060 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.409023    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b74a19be-f798-42d0-88be-dfcc26e526f3-model-cache\") pod \"autores-model-b7292e-kserve-76f965587-9gshm\" (UID: \"b74a19be-f798-42d0-88be-dfcc26e526f3\") " pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:30:51.409115 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.409077    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b74a19be-f798-42d0-88be-dfcc26e526f3-tmp-dir\") pod \"autores-model-b7292e-kserve-76f965587-9gshm\" (UID: \"b74a19be-f798-42d0-88be-dfcc26e526f3\") " pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:30:51.410881 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.410860    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b74a19be-f798-42d0-88be-dfcc26e526f3-dshm\") pod \"autores-model-b7292e-kserve-76f965587-9gshm\" (UID: \"b74a19be-f798-42d0-88be-dfcc26e526f3\") " pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:30:51.411167 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.411152    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b74a19be-f798-42d0-88be-dfcc26e526f3-tls-certs\") pod \"autores-model-b7292e-kserve-76f965587-9gshm\" (UID: \"b74a19be-f798-42d0-88be-dfcc26e526f3\") " pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:30:51.431385 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.431355    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr5rp\" (UniqueName: \"kubernetes.io/projected/b74a19be-f798-42d0-88be-dfcc26e526f3-kube-api-access-wr5rp\") pod \"autores-model-b7292e-kserve-76f965587-9gshm\" (UID: \"b74a19be-f798-42d0-88be-dfcc26e526f3\") " pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:30:51.508226 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.508131    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:30:51.791104 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.791004    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"]
Sep 18 13:30:51.796188 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:30:51.796149    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb74a19be_f798_42d0_88be_dfcc26e526f3.slice/crio-06c8226a8d16ef2c4f8d9042599021c5428467d9327b719babc75c96f13a8adb WatchSource:0}: Error finding container 06c8226a8d16ef2c4f8d9042599021c5428467d9327b719babc75c96f13a8adb: Status 404 returned error can't find the container with id 06c8226a8d16ef2c4f8d9042599021c5428467d9327b719babc75c96f13a8adb
Sep 18 13:30:51.983370 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.983329    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm" event={"ID":"b74a19be-f798-42d0-88be-dfcc26e526f3","Type":"ContainerStarted","Data":"4ee8af0917a1ab889ecd07e7452c01143588fe57ebc1d9ae638ac38d2ebb703a"}
Sep 18 13:30:51.983370 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:51.983374    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm" event={"ID":"b74a19be-f798-42d0-88be-dfcc26e526f3","Type":"ContainerStarted","Data":"06c8226a8d16ef2c4f8d9042599021c5428467d9327b719babc75c96f13a8adb"}
Sep 18 13:30:52.007845 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:52.007788    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm" podStartSLOduration=1.007766886 podStartE2EDuration="1.007766886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:30:52.006508413 +0000 UTC m=+949.361968238" watchObservedRunningTime="2026-09-18 13:30:52.007766886 +0000 UTC m=+949.363226712"
Sep 18 13:30:56.974920 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:30:56.974877    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:31:01.508917 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:01.508877    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:31:01.509362 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:01.508943    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:31:01.524342 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:01.524309    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:31:02.060714 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:02.060677    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:31:04.612850 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:04.612811    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j"]
Sep 18 13:31:04.613300 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:04.613076    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j" podUID="750e0078-8a09-40d3-b543-3bcf7fe984b1" containerName="payload-pre-processing" containerID="cri-o://42728103f470d72e99e218dfb3747d805119f0c9c3b1431b273979a411f90ad9" gracePeriod=30
Sep 18 13:31:04.880741 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:04.880717    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j"
Sep 18 13:31:04.932165 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:04.932130    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msrk6\" (UniqueName: \"kubernetes.io/projected/750e0078-8a09-40d3-b543-3bcf7fe984b1-kube-api-access-msrk6\") pod \"750e0078-8a09-40d3-b543-3bcf7fe984b1\" (UID: \"750e0078-8a09-40d3-b543-3bcf7fe984b1\") "
Sep 18 13:31:04.932316 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:04.932243    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/750e0078-8a09-40d3-b543-3bcf7fe984b1-plugins-config-volume\") pod \"750e0078-8a09-40d3-b543-3bcf7fe984b1\" (UID: \"750e0078-8a09-40d3-b543-3bcf7fe984b1\") "
Sep 18 13:31:04.932624 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:04.932603    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/750e0078-8a09-40d3-b543-3bcf7fe984b1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "750e0078-8a09-40d3-b543-3bcf7fe984b1" (UID: "750e0078-8a09-40d3-b543-3bcf7fe984b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:31:04.934315 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:04.934294    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/750e0078-8a09-40d3-b543-3bcf7fe984b1-kube-api-access-msrk6" (OuterVolumeSpecName: "kube-api-access-msrk6") pod "750e0078-8a09-40d3-b543-3bcf7fe984b1" (UID: "750e0078-8a09-40d3-b543-3bcf7fe984b1"). InnerVolumeSpecName "kube-api-access-msrk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:31:05.033802 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:05.033765    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msrk6\" (UniqueName: \"kubernetes.io/projected/750e0078-8a09-40d3-b543-3bcf7fe984b1-kube-api-access-msrk6\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:31:05.033802 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:05.033795    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/750e0078-8a09-40d3-b543-3bcf7fe984b1-plugins-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:31:05.063733 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:05.063697    2586 generic.go:356] "Generic (PLEG): container finished" podID="750e0078-8a09-40d3-b543-3bcf7fe984b1" containerID="42728103f470d72e99e218dfb3747d805119f0c9c3b1431b273979a411f90ad9" exitCode=0
Sep 18 13:31:05.063891 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:05.063741    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j"
Sep 18 13:31:05.063891 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:05.063777    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j" event={"ID":"750e0078-8a09-40d3-b543-3bcf7fe984b1","Type":"ContainerDied","Data":"42728103f470d72e99e218dfb3747d805119f0c9c3b1431b273979a411f90ad9"}
Sep 18 13:31:05.063891 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:05.063827    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j" event={"ID":"750e0078-8a09-40d3-b543-3bcf7fe984b1","Type":"ContainerDied","Data":"48c37afa2760000f087022f6eb37e6250259147266a42c636ca64ea38f32b237"}
Sep 18 13:31:05.063891 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:05.063851    2586 scope.go:117] "RemoveContainer" containerID="42728103f470d72e99e218dfb3747d805119f0c9c3b1431b273979a411f90ad9"
Sep 18 13:31:05.072822 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:05.072804    2586 scope.go:117] "RemoveContainer" containerID="42728103f470d72e99e218dfb3747d805119f0c9c3b1431b273979a411f90ad9"
Sep 18 13:31:05.073112 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:31:05.073090    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42728103f470d72e99e218dfb3747d805119f0c9c3b1431b273979a411f90ad9\": container with ID starting with 42728103f470d72e99e218dfb3747d805119f0c9c3b1431b273979a411f90ad9 not found: ID does not exist" containerID="42728103f470d72e99e218dfb3747d805119f0c9c3b1431b273979a411f90ad9"
Sep 18 13:31:05.073172 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:05.073121    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42728103f470d72e99e218dfb3747d805119f0c9c3b1431b273979a411f90ad9"} err="failed to get container status \"42728103f470d72e99e218dfb3747d805119f0c9c3b1431b273979a411f90ad9\": rpc error: code = NotFound desc = could not find container \"42728103f470d72e99e218dfb3747d805119f0c9c3b1431b273979a411f90ad9\": container with ID starting with 42728103f470d72e99e218dfb3747d805119f0c9c3b1431b273979a411f90ad9 not found: ID does not exist"
Sep 18 13:31:05.085760 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:05.085739    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j"]
Sep 18 13:31:05.090727 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:05.090707    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-92246a6e-697bcf4cd6-df58j"]
Sep 18 13:31:06.856687 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:06.856657    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="750e0078-8a09-40d3-b543-3bcf7fe984b1" path="/var/lib/kubelet/pods/750e0078-8a09-40d3-b543-3bcf7fe984b1/volumes"
Sep 18 13:31:11.427918 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:11.427870    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45"]
Sep 18 13:31:11.428287 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:11.428274    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="750e0078-8a09-40d3-b543-3bcf7fe984b1" containerName="payload-pre-processing"
Sep 18 13:31:11.428287 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:11.428286    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="750e0078-8a09-40d3-b543-3bcf7fe984b1" containerName="payload-pre-processing"
Sep 18 13:31:11.428389 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:11.428346    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="750e0078-8a09-40d3-b543-3bcf7fe984b1" containerName="payload-pre-processing"
Sep 18 13:31:11.432661 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:11.432642    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45"
Sep 18 13:31:11.435493 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:11.435474    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-cc741da4-dockercfg-69m2k\""
Sep 18 13:31:11.437078 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:11.437059    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-cc741da4\""
Sep 18 13:31:11.448237 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:11.448211    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45"]
Sep 18 13:31:11.496162 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:11.496117    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bc354b-c25f-48b0-bb10-1b1a389e72e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45\" (UID: \"f7bc354b-c25f-48b0-bb10-1b1a389e72e2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45"
Sep 18 13:31:11.496348 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:11.496198    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2snm\" (UniqueName: \"kubernetes.io/projected/f7bc354b-c25f-48b0-bb10-1b1a389e72e2-kube-api-access-c2snm\") pod \"payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45\" (UID: \"f7bc354b-c25f-48b0-bb10-1b1a389e72e2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45"
Sep 18 13:31:11.525668 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:11.525617    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9"]
Sep 18 13:31:11.529249 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:11.529232    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9"
Sep 18 13:31:11.541072 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:11.541046    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9"]
Sep 18 13:31:11.597608 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:11.597570    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bc354b-c25f-48b0-bb10-1b1a389e72e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45\" (UID: \"f7bc354b-c25f-48b0-bb10-1b1a389e72e2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45"
Sep 18 13:31:11.597771 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:11.597617    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb2ef83e-4675-472d-9dc8-59a222f27d52-plugins-config-volume\") pod \"payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9\" (UID: \"eb2ef83e-4675-472d-9dc8-59a222f27d52\") " pod="openshift-ingress/payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9"
Sep 18 13:31:11.597771 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:11.597659    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2snm\" (UniqueName: \"kubernetes.io/projected/f7bc354b-c25f-48b0-bb10-1b1a389e72e2-kube-api-access-c2snm\") pod \"payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45\" (UID: \"f7bc354b-c25f-48b0-bb10-1b1a389e72e2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45"
Sep 18 13:31:11.597771 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:11.597688    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8gkz\" (UniqueName: \"kubernetes.io/projected/eb2ef83e-4675-472d-9dc8-59a222f27d52-kube-api-access-x8gkz\") pod \"payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9\" (UID: \"eb2ef83e-4675-472d-9dc8-59a222f27d52\") " pod="openshift-ingress/payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9"
Sep 18 13:31:11.598641 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:11.598615    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bc354b-c25f-48b0-bb10-1b1a389e72e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45\" (UID: \"f7bc354b-c25f-48b0-bb10-1b1a389e72e2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45"
Sep 18 13:31:11.606945 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:11.606897    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2snm\" (UniqueName: \"kubernetes.io/projected/f7bc354b-c25f-48b0-bb10-1b1a389e72e2-kube-api-access-c2snm\") pod \"payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45\" (UID: \"f7bc354b-c25f-48b0-bb10-1b1a389e72e2\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45"
Sep 18 13:31:11.699091 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:11.699001    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb2ef83e-4675-472d-9dc8-59a222f27d52-plugins-config-volume\") pod \"payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9\" (UID: \"eb2ef83e-4675-472d-9dc8-59a222f27d52\") " pod="openshift-ingress/payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9"
Sep 18 13:31:11.699091 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:11.699084    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8gkz\" (UniqueName: \"kubernetes.io/projected/eb2ef83e-4675-472d-9dc8-59a222f27d52-kube-api-access-x8gkz\") pod \"payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9\" (UID: \"eb2ef83e-4675-472d-9dc8-59a222f27d52\") " pod="openshift-ingress/payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9"
Sep 18 13:31:11.699799 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:11.699779    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb2ef83e-4675-472d-9dc8-59a222f27d52-plugins-config-volume\") pod \"payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9\" (UID: \"eb2ef83e-4675-472d-9dc8-59a222f27d52\") " pod="openshift-ingress/payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9"
Sep 18 13:31:11.708540 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:11.708508    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8gkz\" (UniqueName: \"kubernetes.io/projected/eb2ef83e-4675-472d-9dc8-59a222f27d52-kube-api-access-x8gkz\") pod \"payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9\" (UID: \"eb2ef83e-4675-472d-9dc8-59a222f27d52\") " pod="openshift-ingress/payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9"
Sep 18 13:31:11.742454 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:11.742413    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45"
Sep 18 13:31:11.839799 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:11.839766    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9"
Sep 18 13:31:12.037847 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:12.037811    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45"]
Sep 18 13:31:12.038032 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:31:12.038010    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bc354b_c25f_48b0_bb10_1b1a389e72e2.slice/crio-ce7ad3b2062ee2ee3c1201311c8da9f9461382f811be9051a2947ead5ae3353d WatchSource:0}: Error finding container ce7ad3b2062ee2ee3c1201311c8da9f9461382f811be9051a2947ead5ae3353d: Status 404 returned error can't find the container with id ce7ad3b2062ee2ee3c1201311c8da9f9461382f811be9051a2947ead5ae3353d
Sep 18 13:31:12.108192 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:12.108117    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45" event={"ID":"f7bc354b-c25f-48b0-bb10-1b1a389e72e2","Type":"ContainerStarted","Data":"ce7ad3b2062ee2ee3c1201311c8da9f9461382f811be9051a2947ead5ae3353d"}
Sep 18 13:31:12.114766 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:12.114679    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9"]
Sep 18 13:31:12.118416 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:31:12.118384    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2ef83e_4675_472d_9dc8_59a222f27d52.slice/crio-68fc26e820e08b884d97e5457f0e11f1db4cffe923e2bb0c2e4febfcec755d2e WatchSource:0}: Error finding container 68fc26e820e08b884d97e5457f0e11f1db4cffe923e2bb0c2e4febfcec755d2e: Status 404 returned error can't find the container with id 68fc26e820e08b884d97e5457f0e11f1db4cffe923e2bb0c2e4febfcec755d2e
Sep 18 13:31:13.115521 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:13.115472    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45" event={"ID":"f7bc354b-c25f-48b0-bb10-1b1a389e72e2","Type":"ContainerStarted","Data":"b2d1f2e0a09e6bbe831383f4e1bacc5ee24f73f5a2c6350d49dea0183a0b2f0e"}
Sep 18 13:31:13.116014 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:13.115576    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45"
Sep 18 13:31:13.117230 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:13.117203    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9" event={"ID":"eb2ef83e-4675-472d-9dc8-59a222f27d52","Type":"ContainerStarted","Data":"8d226d15d730d8f00e3a513d328d1d4e39195e4915640a0114b2ad03f9d55f37"}
Sep 18 13:31:13.117358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:13.117266    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9"
Sep 18 13:31:13.117358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:13.117285    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9" event={"ID":"eb2ef83e-4675-472d-9dc8-59a222f27d52","Type":"ContainerStarted","Data":"68fc26e820e08b884d97e5457f0e11f1db4cffe923e2bb0c2e4febfcec755d2e"}
Sep 18 13:31:13.135896 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:13.135831    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45" podStartSLOduration=2.135809613 podStartE2EDuration="2.135809613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:31:13.13450136 +0000 UTC m=+970.489961187" watchObservedRunningTime="2026-09-18 13:31:13.135809613 +0000 UTC m=+970.491269439"
Sep 18 13:31:13.154149 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:13.154093    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9" podStartSLOduration=2.154073576 podStartE2EDuration="2.154073576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:31:13.152805839 +0000 UTC m=+970.508265664" watchObservedRunningTime="2026-09-18 13:31:13.154073576 +0000 UTC m=+970.509533418"
Sep 18 13:31:24.124424 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:24.124350    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9"
Sep 18 13:31:24.124779 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:24.124437    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45"
Sep 18 13:31:24.378705 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:24.378623    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:31:28.872102 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:28.872064    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"]
Sep 18 13:31:38.405916 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:38.405852    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b7292e-29828970-qdv8s"]
Sep 18 13:31:38.409629 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:38.409593    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-b7292e-29828970-qdv8s"]
Sep 18 13:31:38.434219 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:38.434181    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc"]
Sep 18 13:31:38.434728 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:38.434690    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc" podUID="d1ec06bc-7c08-42f5-b5b1-493f91be1d15" containerName="payload-processing" containerID="cri-o://2161bf9ed96070b1f5e2ccfda92f3a28308aa18728c309aef5b09bc135aaeb7c" gracePeriod=30
Sep 18 13:31:38.450317 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:38.450286    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt"]
Sep 18 13:31:38.450589 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:38.450539    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt" podUID="407c62f1-1a0c-420b-89e9-f379905eebe0" containerName="payload-pre-processing" containerID="cri-o://a3dac07a64b6481a509b760509399df75437b3400c6bab92fb55094b35ea02ce" gracePeriod=30
Sep 18 13:31:38.709533 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:38.709502    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc"
Sep 18 13:31:38.735744 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:38.735713    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt"
Sep 18 13:31:38.850011 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:38.849980    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0cc7a30-26f3-4833-8373-869af079e533" path="/var/lib/kubelet/pods/e0cc7a30-26f3-4833-8373-869af079e533/volumes"
Sep 18 13:31:38.863445 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:38.863419    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75j52\" (UniqueName: \"kubernetes.io/projected/d1ec06bc-7c08-42f5-b5b1-493f91be1d15-kube-api-access-75j52\") pod \"d1ec06bc-7c08-42f5-b5b1-493f91be1d15\" (UID: \"d1ec06bc-7c08-42f5-b5b1-493f91be1d15\") "
Sep 18 13:31:38.863604 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:38.863507    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407c62f1-1a0c-420b-89e9-f379905eebe0-plugins-config-volume\") pod \"407c62f1-1a0c-420b-89e9-f379905eebe0\" (UID: \"407c62f1-1a0c-420b-89e9-f379905eebe0\") "
Sep 18 13:31:38.863604 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:38.863543    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1ec06bc-7c08-42f5-b5b1-493f91be1d15-plugins-config-volume\") pod \"d1ec06bc-7c08-42f5-b5b1-493f91be1d15\" (UID: \"d1ec06bc-7c08-42f5-b5b1-493f91be1d15\") "
Sep 18 13:31:38.863604 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:38.863562    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xf5l\" (UniqueName: \"kubernetes.io/projected/407c62f1-1a0c-420b-89e9-f379905eebe0-kube-api-access-6xf5l\") pod \"407c62f1-1a0c-420b-89e9-f379905eebe0\" (UID: \"407c62f1-1a0c-420b-89e9-f379905eebe0\") "
Sep 18 13:31:38.863898 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:38.863875    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ec06bc-7c08-42f5-b5b1-493f91be1d15-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d1ec06bc-7c08-42f5-b5b1-493f91be1d15" (UID: "d1ec06bc-7c08-42f5-b5b1-493f91be1d15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:31:38.863898 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:38.863890    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/407c62f1-1a0c-420b-89e9-f379905eebe0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "407c62f1-1a0c-420b-89e9-f379905eebe0" (UID: "407c62f1-1a0c-420b-89e9-f379905eebe0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:31:38.865719 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:38.865693    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ec06bc-7c08-42f5-b5b1-493f91be1d15-kube-api-access-75j52" (OuterVolumeSpecName: "kube-api-access-75j52") pod "d1ec06bc-7c08-42f5-b5b1-493f91be1d15" (UID: "d1ec06bc-7c08-42f5-b5b1-493f91be1d15"). InnerVolumeSpecName "kube-api-access-75j52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:31:38.865810 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:38.865720    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407c62f1-1a0c-420b-89e9-f379905eebe0-kube-api-access-6xf5l" (OuterVolumeSpecName: "kube-api-access-6xf5l") pod "407c62f1-1a0c-420b-89e9-f379905eebe0" (UID: "407c62f1-1a0c-420b-89e9-f379905eebe0"). InnerVolumeSpecName "kube-api-access-6xf5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:31:38.965232 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:38.964991    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1ec06bc-7c08-42f5-b5b1-493f91be1d15-plugins-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:31:38.965232 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:38.965145    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xf5l\" (UniqueName: \"kubernetes.io/projected/407c62f1-1a0c-420b-89e9-f379905eebe0-kube-api-access-6xf5l\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:31:38.965232 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:38.965168    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75j52\" (UniqueName: \"kubernetes.io/projected/d1ec06bc-7c08-42f5-b5b1-493f91be1d15-kube-api-access-75j52\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:31:38.965232 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:38.965179    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407c62f1-1a0c-420b-89e9-f379905eebe0-plugins-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:31:39.252129 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:39.252044    2586 generic.go:356] "Generic (PLEG): container finished" podID="d1ec06bc-7c08-42f5-b5b1-493f91be1d15" containerID="2161bf9ed96070b1f5e2ccfda92f3a28308aa18728c309aef5b09bc135aaeb7c" exitCode=0
Sep 18 13:31:39.252129 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:39.252091    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc"
Sep 18 13:31:39.252365 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:39.252134    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc" event={"ID":"d1ec06bc-7c08-42f5-b5b1-493f91be1d15","Type":"ContainerDied","Data":"2161bf9ed96070b1f5e2ccfda92f3a28308aa18728c309aef5b09bc135aaeb7c"}
Sep 18 13:31:39.252365 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:39.252190    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc" event={"ID":"d1ec06bc-7c08-42f5-b5b1-493f91be1d15","Type":"ContainerDied","Data":"13de007d6669d4cd114ecbc2aa846a14062e2dcbfeebe365dc68eb838cb4d309"}
Sep 18 13:31:39.252365 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:39.252213    2586 scope.go:117] "RemoveContainer" containerID="2161bf9ed96070b1f5e2ccfda92f3a28308aa18728c309aef5b09bc135aaeb7c"
Sep 18 13:31:39.253484 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:39.253464    2586 generic.go:356] "Generic (PLEG): container finished" podID="407c62f1-1a0c-420b-89e9-f379905eebe0" containerID="a3dac07a64b6481a509b760509399df75437b3400c6bab92fb55094b35ea02ce" exitCode=0
Sep 18 13:31:39.253557 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:39.253507    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt"
Sep 18 13:31:39.253557 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:39.253510    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt" event={"ID":"407c62f1-1a0c-420b-89e9-f379905eebe0","Type":"ContainerDied","Data":"a3dac07a64b6481a509b760509399df75437b3400c6bab92fb55094b35ea02ce"}
Sep 18 13:31:39.253557 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:39.253547    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt" event={"ID":"407c62f1-1a0c-420b-89e9-f379905eebe0","Type":"ContainerDied","Data":"f8eb0d10c7f26aac25bdc122a954da524be8a234e4baa9189c5088b7cc85f054"}
Sep 18 13:31:39.261367 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:39.261342    2586 scope.go:117] "RemoveContainer" containerID="2161bf9ed96070b1f5e2ccfda92f3a28308aa18728c309aef5b09bc135aaeb7c"
Sep 18 13:31:39.261795 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:31:39.261770    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2161bf9ed96070b1f5e2ccfda92f3a28308aa18728c309aef5b09bc135aaeb7c\": container with ID starting with 2161bf9ed96070b1f5e2ccfda92f3a28308aa18728c309aef5b09bc135aaeb7c not found: ID does not exist" containerID="2161bf9ed96070b1f5e2ccfda92f3a28308aa18728c309aef5b09bc135aaeb7c"
Sep 18 13:31:39.261870 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:39.261807    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2161bf9ed96070b1f5e2ccfda92f3a28308aa18728c309aef5b09bc135aaeb7c"} err="failed to get container status \"2161bf9ed96070b1f5e2ccfda92f3a28308aa18728c309aef5b09bc135aaeb7c\": rpc error: code = NotFound desc = could not find container \"2161bf9ed96070b1f5e2ccfda92f3a28308aa18728c309aef5b09bc135aaeb7c\": container with ID starting with 2161bf9ed96070b1f5e2ccfda92f3a28308aa18728c309aef5b09bc135aaeb7c not found: ID does not exist"
Sep 18 13:31:39.261870 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:39.261834    2586 scope.go:117] "RemoveContainer" containerID="a3dac07a64b6481a509b760509399df75437b3400c6bab92fb55094b35ea02ce"
Sep 18 13:31:39.269793 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:39.269771    2586 scope.go:117] "RemoveContainer" containerID="a3dac07a64b6481a509b760509399df75437b3400c6bab92fb55094b35ea02ce"
Sep 18 13:31:39.270070 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:31:39.270053    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3dac07a64b6481a509b760509399df75437b3400c6bab92fb55094b35ea02ce\": container with ID starting with a3dac07a64b6481a509b760509399df75437b3400c6bab92fb55094b35ea02ce not found: ID does not exist" containerID="a3dac07a64b6481a509b760509399df75437b3400c6bab92fb55094b35ea02ce"
Sep 18 13:31:39.270124 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:39.270076    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3dac07a64b6481a509b760509399df75437b3400c6bab92fb55094b35ea02ce"} err="failed to get container status \"a3dac07a64b6481a509b760509399df75437b3400c6bab92fb55094b35ea02ce\": rpc error: code = NotFound desc = could not find container \"a3dac07a64b6481a509b760509399df75437b3400c6bab92fb55094b35ea02ce\": container with ID starting with a3dac07a64b6481a509b760509399df75437b3400c6bab92fb55094b35ea02ce not found: ID does not exist"
Sep 18 13:31:39.300030 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:39.300002    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt"]
Sep 18 13:31:39.316720 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:39.316691    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b7292e-5558798d78-g67bt"]
Sep 18 13:31:39.328312 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:39.328281    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc"]
Sep 18 13:31:39.338567 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:39.338540    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b7292e-5fd7855579-kpdsc"]
Sep 18 13:31:40.851404 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:40.851375    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="407c62f1-1a0c-420b-89e9-f379905eebe0" path="/var/lib/kubelet/pods/407c62f1-1a0c-420b-89e9-f379905eebe0/volumes"
Sep 18 13:31:40.851773 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:40.851756    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1ec06bc-7c08-42f5-b5b1-493f91be1d15" path="/var/lib/kubelet/pods/d1ec06bc-7c08-42f5-b5b1-493f91be1d15/volumes"
Sep 18 13:31:43.891722 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:43.891672    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm" podUID="b74a19be-f798-42d0-88be-dfcc26e526f3" containerName="main" containerID="cri-o://4ee8af0917a1ab889ecd07e7452c01143588fe57ebc1d9ae638ac38d2ebb703a" gracePeriod=45
Sep 18 13:31:44.143545 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.143479    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:31:44.213585 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.213553    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b74a19be-f798-42d0-88be-dfcc26e526f3-dshm\") pod \"b74a19be-f798-42d0-88be-dfcc26e526f3\" (UID: \"b74a19be-f798-42d0-88be-dfcc26e526f3\") "
Sep 18 13:31:44.213585 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.213591    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr5rp\" (UniqueName: \"kubernetes.io/projected/b74a19be-f798-42d0-88be-dfcc26e526f3-kube-api-access-wr5rp\") pod \"b74a19be-f798-42d0-88be-dfcc26e526f3\" (UID: \"b74a19be-f798-42d0-88be-dfcc26e526f3\") "
Sep 18 13:31:44.213825 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.213649    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b74a19be-f798-42d0-88be-dfcc26e526f3-model-cache\") pod \"b74a19be-f798-42d0-88be-dfcc26e526f3\" (UID: \"b74a19be-f798-42d0-88be-dfcc26e526f3\") "
Sep 18 13:31:44.213825 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.213686    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b74a19be-f798-42d0-88be-dfcc26e526f3-home\") pod \"b74a19be-f798-42d0-88be-dfcc26e526f3\" (UID: \"b74a19be-f798-42d0-88be-dfcc26e526f3\") "
Sep 18 13:31:44.213825 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.213763    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b74a19be-f798-42d0-88be-dfcc26e526f3-tls-certs\") pod \"b74a19be-f798-42d0-88be-dfcc26e526f3\" (UID: \"b74a19be-f798-42d0-88be-dfcc26e526f3\") "
Sep 18 13:31:44.213825 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.213809    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b74a19be-f798-42d0-88be-dfcc26e526f3-tmp-dir\") pod \"b74a19be-f798-42d0-88be-dfcc26e526f3\" (UID: \"b74a19be-f798-42d0-88be-dfcc26e526f3\") "
Sep 18 13:31:44.214067 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.213947    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74a19be-f798-42d0-88be-dfcc26e526f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "b74a19be-f798-42d0-88be-dfcc26e526f3" (UID: "b74a19be-f798-42d0-88be-dfcc26e526f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:31:44.214067 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.213996    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74a19be-f798-42d0-88be-dfcc26e526f3-home" (OuterVolumeSpecName: "home") pod "b74a19be-f798-42d0-88be-dfcc26e526f3" (UID: "b74a19be-f798-42d0-88be-dfcc26e526f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:31:44.214208 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.214189    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b74a19be-f798-42d0-88be-dfcc26e526f3-model-cache\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:31:44.214264 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.214220    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b74a19be-f798-42d0-88be-dfcc26e526f3-home\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:31:44.214264 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.214167    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74a19be-f798-42d0-88be-dfcc26e526f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b74a19be-f798-42d0-88be-dfcc26e526f3" (UID: "b74a19be-f798-42d0-88be-dfcc26e526f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:31:44.216139 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.216111    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b74a19be-f798-42d0-88be-dfcc26e526f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b74a19be-f798-42d0-88be-dfcc26e526f3" (UID: "b74a19be-f798-42d0-88be-dfcc26e526f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:31:44.216269 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.216144    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74a19be-f798-42d0-88be-dfcc26e526f3-kube-api-access-wr5rp" (OuterVolumeSpecName: "kube-api-access-wr5rp") pod "b74a19be-f798-42d0-88be-dfcc26e526f3" (UID: "b74a19be-f798-42d0-88be-dfcc26e526f3"). InnerVolumeSpecName "kube-api-access-wr5rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:31:44.216269 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.216162    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74a19be-f798-42d0-88be-dfcc26e526f3-dshm" (OuterVolumeSpecName: "dshm") pod "b74a19be-f798-42d0-88be-dfcc26e526f3" (UID: "b74a19be-f798-42d0-88be-dfcc26e526f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:31:44.282498 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.282464    2586 generic.go:356] "Generic (PLEG): container finished" podID="b74a19be-f798-42d0-88be-dfcc26e526f3" containerID="4ee8af0917a1ab889ecd07e7452c01143588fe57ebc1d9ae638ac38d2ebb703a" exitCode=0
Sep 18 13:31:44.282738 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.282518    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm" event={"ID":"b74a19be-f798-42d0-88be-dfcc26e526f3","Type":"ContainerDied","Data":"4ee8af0917a1ab889ecd07e7452c01143588fe57ebc1d9ae638ac38d2ebb703a"}
Sep 18 13:31:44.282738 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.282532    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"
Sep 18 13:31:44.282738 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.282553    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm" event={"ID":"b74a19be-f798-42d0-88be-dfcc26e526f3","Type":"ContainerDied","Data":"06c8226a8d16ef2c4f8d9042599021c5428467d9327b719babc75c96f13a8adb"}
Sep 18 13:31:44.282738 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.282573    2586 scope.go:117] "RemoveContainer" containerID="4ee8af0917a1ab889ecd07e7452c01143588fe57ebc1d9ae638ac38d2ebb703a"
Sep 18 13:31:44.294299 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.294272    2586 scope.go:117] "RemoveContainer" containerID="4ee8af0917a1ab889ecd07e7452c01143588fe57ebc1d9ae638ac38d2ebb703a"
Sep 18 13:31:44.294693 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:31:44.294665    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ee8af0917a1ab889ecd07e7452c01143588fe57ebc1d9ae638ac38d2ebb703a\": container with ID starting with 4ee8af0917a1ab889ecd07e7452c01143588fe57ebc1d9ae638ac38d2ebb703a not found: ID does not exist" containerID="4ee8af0917a1ab889ecd07e7452c01143588fe57ebc1d9ae638ac38d2ebb703a"
Sep 18 13:31:44.294775 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.294704    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ee8af0917a1ab889ecd07e7452c01143588fe57ebc1d9ae638ac38d2ebb703a"} err="failed to get container status \"4ee8af0917a1ab889ecd07e7452c01143588fe57ebc1d9ae638ac38d2ebb703a\": rpc error: code = NotFound desc = could not find container \"4ee8af0917a1ab889ecd07e7452c01143588fe57ebc1d9ae638ac38d2ebb703a\": container with ID starting with 4ee8af0917a1ab889ecd07e7452c01143588fe57ebc1d9ae638ac38d2ebb703a not found: ID does not exist"
Sep 18 13:31:44.315351 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.315313    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b74a19be-f798-42d0-88be-dfcc26e526f3-tls-certs\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:31:44.315351 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.315337    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b74a19be-f798-42d0-88be-dfcc26e526f3-tmp-dir\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:31:44.315351 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.315349    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b74a19be-f798-42d0-88be-dfcc26e526f3-dshm\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:31:44.315562 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.315361    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr5rp\" (UniqueName: \"kubernetes.io/projected/b74a19be-f798-42d0-88be-dfcc26e526f3-kube-api-access-wr5rp\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:31:44.360408 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.360365    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"]
Sep 18 13:31:44.369449 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.369420    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-b7292e/autores-model-b7292e-kserve-76f965587-9gshm"]
Sep 18 13:31:44.479338 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.479256    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9"]
Sep 18 13:31:44.479557 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.479526    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9" podUID="eb2ef83e-4675-472d-9dc8-59a222f27d52" containerName="payload-processing" containerID="cri-o://8d226d15d730d8f00e3a513d328d1d4e39195e4915640a0114b2ad03f9d55f37" gracePeriod=30
Sep 18 13:31:44.493378 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.493350    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45"]
Sep 18 13:31:44.493554 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.493533    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45" podUID="f7bc354b-c25f-48b0-bb10-1b1a389e72e2" containerName="payload-pre-processing" containerID="cri-o://b2d1f2e0a09e6bbe831383f4e1bacc5ee24f73f5a2c6350d49dea0183a0b2f0e" gracePeriod=30
Sep 18 13:31:44.735355 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.735292    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9"
Sep 18 13:31:44.767423 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.767387    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45"
Sep 18 13:31:44.820994 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.820951    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8gkz\" (UniqueName: \"kubernetes.io/projected/eb2ef83e-4675-472d-9dc8-59a222f27d52-kube-api-access-x8gkz\") pod \"eb2ef83e-4675-472d-9dc8-59a222f27d52\" (UID: \"eb2ef83e-4675-472d-9dc8-59a222f27d52\") "
Sep 18 13:31:44.821196 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.821031    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2snm\" (UniqueName: \"kubernetes.io/projected/f7bc354b-c25f-48b0-bb10-1b1a389e72e2-kube-api-access-c2snm\") pod \"f7bc354b-c25f-48b0-bb10-1b1a389e72e2\" (UID: \"f7bc354b-c25f-48b0-bb10-1b1a389e72e2\") "
Sep 18 13:31:44.821196 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.821101    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bc354b-c25f-48b0-bb10-1b1a389e72e2-plugins-config-volume\") pod \"f7bc354b-c25f-48b0-bb10-1b1a389e72e2\" (UID: \"f7bc354b-c25f-48b0-bb10-1b1a389e72e2\") "
Sep 18 13:31:44.821314 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.821203    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb2ef83e-4675-472d-9dc8-59a222f27d52-plugins-config-volume\") pod \"eb2ef83e-4675-472d-9dc8-59a222f27d52\" (UID: \"eb2ef83e-4675-472d-9dc8-59a222f27d52\") "
Sep 18 13:31:44.821566 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.821537    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7bc354b-c25f-48b0-bb10-1b1a389e72e2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f7bc354b-c25f-48b0-bb10-1b1a389e72e2" (UID: "f7bc354b-c25f-48b0-bb10-1b1a389e72e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:31:44.821566 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.821544    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb2ef83e-4675-472d-9dc8-59a222f27d52-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb2ef83e-4675-472d-9dc8-59a222f27d52" (UID: "eb2ef83e-4675-472d-9dc8-59a222f27d52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:31:44.823327 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.823303    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7bc354b-c25f-48b0-bb10-1b1a389e72e2-kube-api-access-c2snm" (OuterVolumeSpecName: "kube-api-access-c2snm") pod "f7bc354b-c25f-48b0-bb10-1b1a389e72e2" (UID: "f7bc354b-c25f-48b0-bb10-1b1a389e72e2"). InnerVolumeSpecName "kube-api-access-c2snm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:31:44.823438 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.823413    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb2ef83e-4675-472d-9dc8-59a222f27d52-kube-api-access-x8gkz" (OuterVolumeSpecName: "kube-api-access-x8gkz") pod "eb2ef83e-4675-472d-9dc8-59a222f27d52" (UID: "eb2ef83e-4675-472d-9dc8-59a222f27d52"). InnerVolumeSpecName "kube-api-access-x8gkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:31:44.849578 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.849546    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b74a19be-f798-42d0-88be-dfcc26e526f3" path="/var/lib/kubelet/pods/b74a19be-f798-42d0-88be-dfcc26e526f3/volumes"
Sep 18 13:31:44.922219 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.922186    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb2ef83e-4675-472d-9dc8-59a222f27d52-plugins-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:31:44.922219 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.922217    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8gkz\" (UniqueName: \"kubernetes.io/projected/eb2ef83e-4675-472d-9dc8-59a222f27d52-kube-api-access-x8gkz\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:31:44.922591 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.922231    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2snm\" (UniqueName: \"kubernetes.io/projected/f7bc354b-c25f-48b0-bb10-1b1a389e72e2-kube-api-access-c2snm\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:31:44.922591 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:44.922243    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bc354b-c25f-48b0-bb10-1b1a389e72e2-plugins-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:31:45.288518 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:45.288482    2586 generic.go:356] "Generic (PLEG): container finished" podID="eb2ef83e-4675-472d-9dc8-59a222f27d52" containerID="8d226d15d730d8f00e3a513d328d1d4e39195e4915640a0114b2ad03f9d55f37" exitCode=0
Sep 18 13:31:45.288730 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:45.288537    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9"
Sep 18 13:31:45.288730 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:45.288575    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9" event={"ID":"eb2ef83e-4675-472d-9dc8-59a222f27d52","Type":"ContainerDied","Data":"8d226d15d730d8f00e3a513d328d1d4e39195e4915640a0114b2ad03f9d55f37"}
Sep 18 13:31:45.288730 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:45.288634    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9" event={"ID":"eb2ef83e-4675-472d-9dc8-59a222f27d52","Type":"ContainerDied","Data":"68fc26e820e08b884d97e5457f0e11f1db4cffe923e2bb0c2e4febfcec755d2e"}
Sep 18 13:31:45.288730 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:45.288664    2586 scope.go:117] "RemoveContainer" containerID="8d226d15d730d8f00e3a513d328d1d4e39195e4915640a0114b2ad03f9d55f37"
Sep 18 13:31:45.291194 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:45.291168    2586 generic.go:356] "Generic (PLEG): container finished" podID="f7bc354b-c25f-48b0-bb10-1b1a389e72e2" containerID="b2d1f2e0a09e6bbe831383f4e1bacc5ee24f73f5a2c6350d49dea0183a0b2f0e" exitCode=0
Sep 18 13:31:45.291323 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:45.291228    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45" event={"ID":"f7bc354b-c25f-48b0-bb10-1b1a389e72e2","Type":"ContainerDied","Data":"b2d1f2e0a09e6bbe831383f4e1bacc5ee24f73f5a2c6350d49dea0183a0b2f0e"}
Sep 18 13:31:45.291323 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:45.291258    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45"
Sep 18 13:31:45.291323 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:45.291269    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45" event={"ID":"f7bc354b-c25f-48b0-bb10-1b1a389e72e2","Type":"ContainerDied","Data":"ce7ad3b2062ee2ee3c1201311c8da9f9461382f811be9051a2947ead5ae3353d"}
Sep 18 13:31:45.297926 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:45.297883    2586 scope.go:117] "RemoveContainer" containerID="8d226d15d730d8f00e3a513d328d1d4e39195e4915640a0114b2ad03f9d55f37"
Sep 18 13:31:45.298347 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:31:45.298327    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d226d15d730d8f00e3a513d328d1d4e39195e4915640a0114b2ad03f9d55f37\": container with ID starting with 8d226d15d730d8f00e3a513d328d1d4e39195e4915640a0114b2ad03f9d55f37 not found: ID does not exist" containerID="8d226d15d730d8f00e3a513d328d1d4e39195e4915640a0114b2ad03f9d55f37"
Sep 18 13:31:45.298424 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:45.298355    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d226d15d730d8f00e3a513d328d1d4e39195e4915640a0114b2ad03f9d55f37"} err="failed to get container status \"8d226d15d730d8f00e3a513d328d1d4e39195e4915640a0114b2ad03f9d55f37\": rpc error: code = NotFound desc = could not find container \"8d226d15d730d8f00e3a513d328d1d4e39195e4915640a0114b2ad03f9d55f37\": container with ID starting with 8d226d15d730d8f00e3a513d328d1d4e39195e4915640a0114b2ad03f9d55f37 not found: ID does not exist"
Sep 18 13:31:45.298424 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:45.298374    2586 scope.go:117] "RemoveContainer" containerID="b2d1f2e0a09e6bbe831383f4e1bacc5ee24f73f5a2c6350d49dea0183a0b2f0e"
Sep 18 13:31:45.307864 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:45.307834    2586 scope.go:117] "RemoveContainer" containerID="b2d1f2e0a09e6bbe831383f4e1bacc5ee24f73f5a2c6350d49dea0183a0b2f0e"
Sep 18 13:31:45.308256 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:31:45.308232    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2d1f2e0a09e6bbe831383f4e1bacc5ee24f73f5a2c6350d49dea0183a0b2f0e\": container with ID starting with b2d1f2e0a09e6bbe831383f4e1bacc5ee24f73f5a2c6350d49dea0183a0b2f0e not found: ID does not exist" containerID="b2d1f2e0a09e6bbe831383f4e1bacc5ee24f73f5a2c6350d49dea0183a0b2f0e"
Sep 18 13:31:45.308327 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:45.308268    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2d1f2e0a09e6bbe831383f4e1bacc5ee24f73f5a2c6350d49dea0183a0b2f0e"} err="failed to get container status \"b2d1f2e0a09e6bbe831383f4e1bacc5ee24f73f5a2c6350d49dea0183a0b2f0e\": rpc error: code = NotFound desc = could not find container \"b2d1f2e0a09e6bbe831383f4e1bacc5ee24f73f5a2c6350d49dea0183a0b2f0e\": container with ID starting with b2d1f2e0a09e6bbe831383f4e1bacc5ee24f73f5a2c6350d49dea0183a0b2f0e not found: ID does not exist"
Sep 18 13:31:45.319059 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:45.319018    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9"]
Sep 18 13:31:45.325137 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:45.325106    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cc741da4-57747f4d89-vqvv9"]
Sep 18 13:31:45.347268 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:45.347214    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45"]
Sep 18 13:31:45.350281 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:45.350251    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cc741da4-8849bfdf4-gjs45"]
Sep 18 13:31:46.850147 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:46.850109    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb2ef83e-4675-472d-9dc8-59a222f27d52" path="/var/lib/kubelet/pods/eb2ef83e-4675-472d-9dc8-59a222f27d52/volumes"
Sep 18 13:31:46.850499 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:46.850462    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7bc354b-c25f-48b0-bb10-1b1a389e72e2" path="/var/lib/kubelet/pods/f7bc354b-c25f-48b0-bb10-1b1a389e72e2/volumes"
Sep 18 13:31:49.507544 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:49.507505    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:31:49.999180 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:49.999133    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"]
Sep 18 13:31:49.999698 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:49.999666    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="407c62f1-1a0c-420b-89e9-f379905eebe0" containerName="payload-pre-processing"
Sep 18 13:31:49.999698 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:49.999695    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="407c62f1-1a0c-420b-89e9-f379905eebe0" containerName="payload-pre-processing"
Sep 18 13:31:49.999882 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:49.999709    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1ec06bc-7c08-42f5-b5b1-493f91be1d15" containerName="payload-processing"
Sep 18 13:31:49.999882 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:49.999717    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1ec06bc-7c08-42f5-b5b1-493f91be1d15" containerName="payload-processing"
Sep 18 13:31:49.999882 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:49.999742    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b74a19be-f798-42d0-88be-dfcc26e526f3" containerName="main"
Sep 18 13:31:49.999882 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:49.999751    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b74a19be-f798-42d0-88be-dfcc26e526f3" containerName="main"
Sep 18 13:31:49.999882 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:49.999763    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb2ef83e-4675-472d-9dc8-59a222f27d52" containerName="payload-processing"
Sep 18 13:31:49.999882 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:49.999771    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb2ef83e-4675-472d-9dc8-59a222f27d52" containerName="payload-processing"
Sep 18 13:31:49.999882 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:49.999796    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7bc354b-c25f-48b0-bb10-1b1a389e72e2" containerName="payload-pre-processing"
Sep 18 13:31:49.999882 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:49.999803    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7bc354b-c25f-48b0-bb10-1b1a389e72e2" containerName="payload-pre-processing"
Sep 18 13:31:50.000272 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:49.999892    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7bc354b-c25f-48b0-bb10-1b1a389e72e2" containerName="payload-pre-processing"
Sep 18 13:31:50.000272 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:49.999941    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b74a19be-f798-42d0-88be-dfcc26e526f3" containerName="main"
Sep 18 13:31:50.000272 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:49.999952    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1ec06bc-7c08-42f5-b5b1-493f91be1d15" containerName="payload-processing"
Sep 18 13:31:50.000272 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:49.999963    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="407c62f1-1a0c-420b-89e9-f379905eebe0" containerName="payload-pre-processing"
Sep 18 13:31:50.000272 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:49.999975    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb2ef83e-4675-472d-9dc8-59a222f27d52" containerName="payload-processing"
Sep 18 13:31:50.003479 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.003453    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.009619 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.009593    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 13:31:50.010861 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.010840    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-808e44-openshift-default-dockercfg-j5srk\""
Sep 18 13:31:50.026703 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.026669    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"]
Sep 18 13:31:50.070621 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.070575    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-credential-socket\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.070811 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.070637    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-workload-certs\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.070811 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.070691    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-workload-socket\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.070811 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.070734    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a0f684d-5cab-42d6-84b1-20cc2470808b-istio-podinfo\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.070811 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.070776    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a0f684d-5cab-42d6-84b1-20cc2470808b-istiod-ca-cert\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.070811 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.070813    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a0f684d-5cab-42d6-84b1-20cc2470808b-istio-token\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.071171 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.070860    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-istio-data\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.071171 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.070915    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-istio-envoy\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.071171 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.070961    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5c5x\" (UniqueName: \"kubernetes.io/projected/2a0f684d-5cab-42d6-84b1-20cc2470808b-kube-api-access-f5c5x\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.171962 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.171896    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a0f684d-5cab-42d6-84b1-20cc2470808b-istio-podinfo\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.171962 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.171967    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a0f684d-5cab-42d6-84b1-20cc2470808b-istiod-ca-cert\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.172233 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.172001    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a0f684d-5cab-42d6-84b1-20cc2470808b-istio-token\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.172233 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.172159    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-istio-data\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.172233 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.172208    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-istio-envoy\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.172397 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.172280    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5c5x\" (UniqueName: \"kubernetes.io/projected/2a0f684d-5cab-42d6-84b1-20cc2470808b-kube-api-access-f5c5x\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.172397 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.172343    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-credential-socket\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.172511 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.172399    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-workload-certs\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.172511 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.172428    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-workload-socket\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.172606 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.172511    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-istio-data\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.172889 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.172798    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a0f684d-5cab-42d6-84b1-20cc2470808b-istiod-ca-cert\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.172889 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.172803    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-workload-socket\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.172889 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.172894    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-workload-certs\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.173135 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.172949    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-credential-socket\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.174618 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.174590    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-istio-envoy\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.174764 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.174748    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a0f684d-5cab-42d6-84b1-20cc2470808b-istio-podinfo\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.195360 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.195325    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5c5x\" (UniqueName: \"kubernetes.io/projected/2a0f684d-5cab-42d6-84b1-20cc2470808b-kube-api-access-f5c5x\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.196821 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.196797    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a0f684d-5cab-42d6-84b1-20cc2470808b-istio-token\") pod \"e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") " pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.317132 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.317053    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:50.638652 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.638617    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"]
Sep 18 13:31:50.641556 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:31:50.641527    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0f684d_5cab_42d6_84b1_20cc2470808b.slice/crio-5dd276b842c0699c551ef0fef33fb1a5d983f4ab7abc491d70cc9628f3776735 WatchSource:0}: Error finding container 5dd276b842c0699c551ef0fef33fb1a5d983f4ab7abc491d70cc9628f3776735: Status 404 returned error can't find the container with id 5dd276b842c0699c551ef0fef33fb1a5d983f4ab7abc491d70cc9628f3776735
Sep 18 13:31:50.643456 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:50.643438    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 13:31:51.327583 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:51.327546    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg" event={"ID":"2a0f684d-5cab-42d6-84b1-20cc2470808b","Type":"ContainerStarted","Data":"5dd276b842c0699c551ef0fef33fb1a5d983f4ab7abc491d70cc9628f3776735"}
Sep 18 13:31:54.093145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:54.093103    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:31:54.093500 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:54.093201    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:31:54.093500 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:54.093250    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:31:54.348183 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:54.348096    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg" event={"ID":"2a0f684d-5cab-42d6-84b1-20cc2470808b","Type":"ContainerStarted","Data":"ac1e77a63be3096441355789528322fb4d634a121d9cd3fe3fbde651cdd1bf9a"}
Sep 18 13:31:54.402242 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:54.402182    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg" podStartSLOduration=1.9530304379999999 podStartE2EDuration="5.402165202s" totalImagesPullingTime="3.449134764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:31:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:31:54.401019901 +0000 UTC m=+1011.756479725" watchObservedRunningTime="2026-09-18 13:31:54.402165202 +0000 UTC m=+1011.757625029"
Sep 18 13:31:55.317987 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:55.317948    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:55.324161 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:55.324134    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:55.353555 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:55.353512    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:31:55.354611 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:31:55.354589    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:32:09.192501 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.192467    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"]
Sep 18 13:32:09.197679 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.197655    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:32:09.200365 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.200338    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5be8a3\"/\"default-dockercfg-b7jf8\""
Sep 18 13:32:09.200508 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.200379    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5be8a3\"/\"premium-w5-5be8a3-kserve-self-signed-certs\""
Sep 18 13:32:09.200648 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.200629    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-5be8a3\"/\"openshift-service-ca.crt\""
Sep 18 13:32:09.200703 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.200636    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-5be8a3\"/\"kube-root-ca.crt\""
Sep 18 13:32:09.210665 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.210637    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"]
Sep 18 13:32:09.364338 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.364301    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eb25909-07c3-49d0-ad0e-207793edcd72-tmp-dir\") pod \"premium-w5-5be8a3-kserve-6456fd6c57-pccb2\" (UID: \"9eb25909-07c3-49d0-ad0e-207793edcd72\") " pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:32:09.364338 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.364343    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eb25909-07c3-49d0-ad0e-207793edcd72-dshm\") pod \"premium-w5-5be8a3-kserve-6456fd6c57-pccb2\" (UID: \"9eb25909-07c3-49d0-ad0e-207793edcd72\") " pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:32:09.364680 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.364364    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eb25909-07c3-49d0-ad0e-207793edcd72-tls-certs\") pod \"premium-w5-5be8a3-kserve-6456fd6c57-pccb2\" (UID: \"9eb25909-07c3-49d0-ad0e-207793edcd72\") " pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:32:09.364680 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.364418    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eb25909-07c3-49d0-ad0e-207793edcd72-home\") pod \"premium-w5-5be8a3-kserve-6456fd6c57-pccb2\" (UID: \"9eb25909-07c3-49d0-ad0e-207793edcd72\") " pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:32:09.364680 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.364455    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmg5k\" (UniqueName: \"kubernetes.io/projected/9eb25909-07c3-49d0-ad0e-207793edcd72-kube-api-access-bmg5k\") pod \"premium-w5-5be8a3-kserve-6456fd6c57-pccb2\" (UID: \"9eb25909-07c3-49d0-ad0e-207793edcd72\") " pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:32:09.364680 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.364507    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eb25909-07c3-49d0-ad0e-207793edcd72-model-cache\") pod \"premium-w5-5be8a3-kserve-6456fd6c57-pccb2\" (UID: \"9eb25909-07c3-49d0-ad0e-207793edcd72\") " pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:32:09.465293 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.465211    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eb25909-07c3-49d0-ad0e-207793edcd72-home\") pod \"premium-w5-5be8a3-kserve-6456fd6c57-pccb2\" (UID: \"9eb25909-07c3-49d0-ad0e-207793edcd72\") " pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:32:09.465293 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.465249    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmg5k\" (UniqueName: \"kubernetes.io/projected/9eb25909-07c3-49d0-ad0e-207793edcd72-kube-api-access-bmg5k\") pod \"premium-w5-5be8a3-kserve-6456fd6c57-pccb2\" (UID: \"9eb25909-07c3-49d0-ad0e-207793edcd72\") " pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:32:09.465293 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.465292    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eb25909-07c3-49d0-ad0e-207793edcd72-model-cache\") pod \"premium-w5-5be8a3-kserve-6456fd6c57-pccb2\" (UID: \"9eb25909-07c3-49d0-ad0e-207793edcd72\") " pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:32:09.465523 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.465421    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eb25909-07c3-49d0-ad0e-207793edcd72-tmp-dir\") pod \"premium-w5-5be8a3-kserve-6456fd6c57-pccb2\" (UID: \"9eb25909-07c3-49d0-ad0e-207793edcd72\") " pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:32:09.465523 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.465452    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eb25909-07c3-49d0-ad0e-207793edcd72-dshm\") pod \"premium-w5-5be8a3-kserve-6456fd6c57-pccb2\" (UID: \"9eb25909-07c3-49d0-ad0e-207793edcd72\") " pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:32:09.465523 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.465482    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eb25909-07c3-49d0-ad0e-207793edcd72-tls-certs\") pod \"premium-w5-5be8a3-kserve-6456fd6c57-pccb2\" (UID: \"9eb25909-07c3-49d0-ad0e-207793edcd72\") " pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:32:09.465725 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.465702    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eb25909-07c3-49d0-ad0e-207793edcd72-home\") pod \"premium-w5-5be8a3-kserve-6456fd6c57-pccb2\" (UID: \"9eb25909-07c3-49d0-ad0e-207793edcd72\") " pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:32:09.465791 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.465725    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eb25909-07c3-49d0-ad0e-207793edcd72-model-cache\") pod \"premium-w5-5be8a3-kserve-6456fd6c57-pccb2\" (UID: \"9eb25909-07c3-49d0-ad0e-207793edcd72\") " pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:32:09.465791 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.465775    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eb25909-07c3-49d0-ad0e-207793edcd72-tmp-dir\") pod \"premium-w5-5be8a3-kserve-6456fd6c57-pccb2\" (UID: \"9eb25909-07c3-49d0-ad0e-207793edcd72\") " pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:32:09.467946 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.467916    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eb25909-07c3-49d0-ad0e-207793edcd72-dshm\") pod \"premium-w5-5be8a3-kserve-6456fd6c57-pccb2\" (UID: \"9eb25909-07c3-49d0-ad0e-207793edcd72\") " pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:32:09.468129 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.468112    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eb25909-07c3-49d0-ad0e-207793edcd72-tls-certs\") pod \"premium-w5-5be8a3-kserve-6456fd6c57-pccb2\" (UID: \"9eb25909-07c3-49d0-ad0e-207793edcd72\") " pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:32:09.473314 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.473294    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmg5k\" (UniqueName: \"kubernetes.io/projected/9eb25909-07c3-49d0-ad0e-207793edcd72-kube-api-access-bmg5k\") pod \"premium-w5-5be8a3-kserve-6456fd6c57-pccb2\" (UID: \"9eb25909-07c3-49d0-ad0e-207793edcd72\") " pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:32:09.478440 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.478418    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:32:09.511197 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.511165    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:32:09.791686 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:09.791660    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"]
Sep 18 13:32:09.794386 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:32:09.794348    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9eb25909_07c3_49d0_ad0e_207793edcd72.slice/crio-0368d04514478da00ea5570bca7c4c7898c9f524a363497f4e883865a357f492 WatchSource:0}: Error finding container 0368d04514478da00ea5570bca7c4c7898c9f524a363497f4e883865a357f492: Status 404 returned error can't find the container with id 0368d04514478da00ea5570bca7c4c7898c9f524a363497f4e883865a357f492
Sep 18 13:32:10.433722 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:10.433685    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2" event={"ID":"9eb25909-07c3-49d0-ad0e-207793edcd72","Type":"ContainerStarted","Data":"aee2d72746984ab542e0c013e5c0b98d8467f62b28ae32d852407bcf9fccb9db"}
Sep 18 13:32:10.434252 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:10.433735    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2" event={"ID":"9eb25909-07c3-49d0-ad0e-207793edcd72","Type":"ContainerStarted","Data":"0368d04514478da00ea5570bca7c4c7898c9f524a363497f4e883865a357f492"}
Sep 18 13:32:10.453365 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:10.453309    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2" podStartSLOduration=1.453293107 podStartE2EDuration="1.453293107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:32:10.451394197 +0000 UTC m=+1027.806854023" watchObservedRunningTime="2026-09-18 13:32:10.453293107 +0000 UTC m=+1027.808752952"
Sep 18 13:32:19.479805 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:19.479764    2586 generic.go:356] "Generic (PLEG): container finished" podID="2a0f684d-5cab-42d6-84b1-20cc2470808b" containerID="ac1e77a63be3096441355789528322fb4d634a121d9cd3fe3fbde651cdd1bf9a" exitCode=0
Sep 18 13:32:19.480216 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:19.479837    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg" event={"ID":"2a0f684d-5cab-42d6-84b1-20cc2470808b","Type":"ContainerDied","Data":"ac1e77a63be3096441355789528322fb4d634a121d9cd3fe3fbde651cdd1bf9a"}
Sep 18 13:32:19.480259 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:19.480233    2586 scope.go:117] "RemoveContainer" containerID="ac1e77a63be3096441355789528322fb4d634a121d9cd3fe3fbde651cdd1bf9a"
Sep 18 13:32:19.481370 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:19.481340    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:32:19.481453 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:19.481416    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:32:19.481453 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:19.481448    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:32:19.511689 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:19.511667    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:32:19.511987 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:19.511945    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:32:19.525893 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:19.525874    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:32:20.487636 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:20.487596    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg" event={"ID":"2a0f684d-5cab-42d6-84b1-20cc2470808b","Type":"ContainerStarted","Data":"cf62afcb3f80e1dd6c726fec80e086607c5b3a001493b5285e789848bda9d447"}
Sep 18 13:32:20.498947 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:20.498895    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:32:21.319239 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:21.319192    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:32:21.319460 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:21.319334    2586 patch_prober.go:28] interesting pod/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 18 13:32:21.319460 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:21.319396    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg" podUID="2a0f684d-5cab-42d6-84b1-20cc2470808b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 18 13:32:22.317725 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:22.317674    2586 patch_prober.go:28] interesting pod/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 18 13:32:22.318257 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:22.317749    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg" podUID="2a0f684d-5cab-42d6-84b1-20cc2470808b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 18 13:32:23.318151 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:23.318112    2586 patch_prober.go:28] interesting pod/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 18 13:32:23.318748 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:23.318716    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg" podUID="2a0f684d-5cab-42d6-84b1-20cc2470808b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 18 13:32:24.318441 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:24.318402    2586 patch_prober.go:28] interesting pod/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 18 13:32:24.319095 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:24.319051    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg" podUID="2a0f684d-5cab-42d6-84b1-20cc2470808b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 18 13:32:25.318513 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:25.318464    2586 patch_prober.go:28] interesting pod/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 18 13:32:25.319038 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:25.318561    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg" podUID="2a0f684d-5cab-42d6-84b1-20cc2470808b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 18 13:32:25.353854 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:25.353814    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:32:26.322429 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:26.322397    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:32:26.323381 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:26.323361    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:32:33.688274 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:33.688237    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:32:38.035928 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.035875    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"]
Sep 18 13:32:38.039507 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.039488    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:32:38.043251 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.043211    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-951469\"/\"kube-root-ca.crt\""
Sep 18 13:32:38.043373 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.043215    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-951469\"/\"premium-w0-951469-kserve-self-signed-certs\""
Sep 18 13:32:38.043373 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.043332    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-951469\"/\"default-dockercfg-lxfdl\""
Sep 18 13:32:38.046489 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.046466    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-951469\"/\"openshift-service-ca.crt\""
Sep 18 13:32:38.072755 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.072716    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"]
Sep 18 13:32:38.137633 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.137594    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce712460-0d49-4b14-ad89-67160b0c1320-tmp-dir\") pod \"premium-w0-951469-kserve-b757b4d6b-9j72w\" (UID: \"ce712460-0d49-4b14-ad89-67160b0c1320\") " pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:32:38.137633 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.137634    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce712460-0d49-4b14-ad89-67160b0c1320-model-cache\") pod \"premium-w0-951469-kserve-b757b4d6b-9j72w\" (UID: \"ce712460-0d49-4b14-ad89-67160b0c1320\") " pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:32:38.137866 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.137670    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce712460-0d49-4b14-ad89-67160b0c1320-dshm\") pod \"premium-w0-951469-kserve-b757b4d6b-9j72w\" (UID: \"ce712460-0d49-4b14-ad89-67160b0c1320\") " pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:32:38.137866 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.137760    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpdlh\" (UniqueName: \"kubernetes.io/projected/ce712460-0d49-4b14-ad89-67160b0c1320-kube-api-access-xpdlh\") pod \"premium-w0-951469-kserve-b757b4d6b-9j72w\" (UID: \"ce712460-0d49-4b14-ad89-67160b0c1320\") " pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:32:38.137866 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.137807    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce712460-0d49-4b14-ad89-67160b0c1320-home\") pod \"premium-w0-951469-kserve-b757b4d6b-9j72w\" (UID: \"ce712460-0d49-4b14-ad89-67160b0c1320\") " pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:32:38.137866 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.137822    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce712460-0d49-4b14-ad89-67160b0c1320-tls-certs\") pod \"premium-w0-951469-kserve-b757b4d6b-9j72w\" (UID: \"ce712460-0d49-4b14-ad89-67160b0c1320\") " pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:32:38.238314 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.238283    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce712460-0d49-4b14-ad89-67160b0c1320-tmp-dir\") pod \"premium-w0-951469-kserve-b757b4d6b-9j72w\" (UID: \"ce712460-0d49-4b14-ad89-67160b0c1320\") " pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:32:38.238314 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.238321    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce712460-0d49-4b14-ad89-67160b0c1320-model-cache\") pod \"premium-w0-951469-kserve-b757b4d6b-9j72w\" (UID: \"ce712460-0d49-4b14-ad89-67160b0c1320\") " pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:32:38.238549 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.238357    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce712460-0d49-4b14-ad89-67160b0c1320-dshm\") pod \"premium-w0-951469-kserve-b757b4d6b-9j72w\" (UID: \"ce712460-0d49-4b14-ad89-67160b0c1320\") " pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:32:38.238549 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.238403    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpdlh\" (UniqueName: \"kubernetes.io/projected/ce712460-0d49-4b14-ad89-67160b0c1320-kube-api-access-xpdlh\") pod \"premium-w0-951469-kserve-b757b4d6b-9j72w\" (UID: \"ce712460-0d49-4b14-ad89-67160b0c1320\") " pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:32:38.238549 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.238433    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce712460-0d49-4b14-ad89-67160b0c1320-home\") pod \"premium-w0-951469-kserve-b757b4d6b-9j72w\" (UID: \"ce712460-0d49-4b14-ad89-67160b0c1320\") " pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:32:38.238549 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.238449    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce712460-0d49-4b14-ad89-67160b0c1320-tls-certs\") pod \"premium-w0-951469-kserve-b757b4d6b-9j72w\" (UID: \"ce712460-0d49-4b14-ad89-67160b0c1320\") " pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:32:38.238764 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.238741    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce712460-0d49-4b14-ad89-67160b0c1320-tmp-dir\") pod \"premium-w0-951469-kserve-b757b4d6b-9j72w\" (UID: \"ce712460-0d49-4b14-ad89-67160b0c1320\") " pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:32:38.238824 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.238778    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce712460-0d49-4b14-ad89-67160b0c1320-model-cache\") pod \"premium-w0-951469-kserve-b757b4d6b-9j72w\" (UID: \"ce712460-0d49-4b14-ad89-67160b0c1320\") " pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:32:38.238824 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.238797    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce712460-0d49-4b14-ad89-67160b0c1320-home\") pod \"premium-w0-951469-kserve-b757b4d6b-9j72w\" (UID: \"ce712460-0d49-4b14-ad89-67160b0c1320\") " pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:32:38.240792 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.240769    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce712460-0d49-4b14-ad89-67160b0c1320-dshm\") pod \"premium-w0-951469-kserve-b757b4d6b-9j72w\" (UID: \"ce712460-0d49-4b14-ad89-67160b0c1320\") " pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:32:38.241046 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.241030    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce712460-0d49-4b14-ad89-67160b0c1320-tls-certs\") pod \"premium-w0-951469-kserve-b757b4d6b-9j72w\" (UID: \"ce712460-0d49-4b14-ad89-67160b0c1320\") " pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:32:38.255735 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.255709    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpdlh\" (UniqueName: \"kubernetes.io/projected/ce712460-0d49-4b14-ad89-67160b0c1320-kube-api-access-xpdlh\") pod \"premium-w0-951469-kserve-b757b4d6b-9j72w\" (UID: \"ce712460-0d49-4b14-ad89-67160b0c1320\") " pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:32:38.349766 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.349682    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:32:38.630355 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:38.630277    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"]
Sep 18 13:32:38.634067 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:32:38.634039    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce712460_0d49_4b14_ad89_67160b0c1320.slice/crio-37c2ee38fdd2e6d36d63b6a1688ba74134c5deb1dc6c02eec2abecd95733f57a WatchSource:0}: Error finding container 37c2ee38fdd2e6d36d63b6a1688ba74134c5deb1dc6c02eec2abecd95733f57a: Status 404 returned error can't find the container with id 37c2ee38fdd2e6d36d63b6a1688ba74134c5deb1dc6c02eec2abecd95733f57a
Sep 18 13:32:39.587280 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:39.587241    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w" event={"ID":"ce712460-0d49-4b14-ad89-67160b0c1320","Type":"ContainerStarted","Data":"7fd8025741f740e374c50cd3c94d1f0ceb70908529badb6c583ce55daa870264"}
Sep 18 13:32:39.587280 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:39.587286    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w" event={"ID":"ce712460-0d49-4b14-ad89-67160b0c1320","Type":"ContainerStarted","Data":"37c2ee38fdd2e6d36d63b6a1688ba74134c5deb1dc6c02eec2abecd95733f57a"}
Sep 18 13:32:39.612567 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:39.612512    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w" podStartSLOduration=1.612490319 podStartE2EDuration="1.612490319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:32:39.612014395 +0000 UTC m=+1056.967474230" watchObservedRunningTime="2026-09-18 13:32:39.612490319 +0000 UTC m=+1056.967950146"
Sep 18 13:32:48.350431 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:48.350340    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:32:48.350897 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:48.350650    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:32:48.363279 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:48.363255    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:32:48.647552 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:48.647521    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:32:57.172888 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:32:57.172854    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:33:18.269880 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.269849    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"]
Sep 18 13:33:18.273767 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.273746    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:33:18.276763 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.276733    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-808e44\"/\"explicit-model-808e44-kserve-self-signed-certs\""
Sep 18 13:33:18.276763 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.276760    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-808e44\"/\"openshift-service-ca.crt\""
Sep 18 13:33:18.277039 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.276774    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-808e44\"/\"kube-root-ca.crt\""
Sep 18 13:33:18.277039 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.276853    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-808e44\"/\"default-dockercfg-vdsh7\""
Sep 18 13:33:18.284021 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.284000    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"]
Sep 18 13:33:18.326700 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.326656    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f565450e-c855-4701-ab57-9c8d29a0e004-tls-certs\") pod \"explicit-model-808e44-kserve-69f85f47b7-crp2h\" (UID: \"f565450e-c855-4701-ab57-9c8d29a0e004\") " pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:33:18.326870 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.326721    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql968\" (UniqueName: \"kubernetes.io/projected/f565450e-c855-4701-ab57-9c8d29a0e004-kube-api-access-ql968\") pod \"explicit-model-808e44-kserve-69f85f47b7-crp2h\" (UID: \"f565450e-c855-4701-ab57-9c8d29a0e004\") " pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:33:18.326870 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.326849    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f565450e-c855-4701-ab57-9c8d29a0e004-home\") pod \"explicit-model-808e44-kserve-69f85f47b7-crp2h\" (UID: \"f565450e-c855-4701-ab57-9c8d29a0e004\") " pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:33:18.326990 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.326880    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f565450e-c855-4701-ab57-9c8d29a0e004-dshm\") pod \"explicit-model-808e44-kserve-69f85f47b7-crp2h\" (UID: \"f565450e-c855-4701-ab57-9c8d29a0e004\") " pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:33:18.326990 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.326928    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f565450e-c855-4701-ab57-9c8d29a0e004-model-cache\") pod \"explicit-model-808e44-kserve-69f85f47b7-crp2h\" (UID: \"f565450e-c855-4701-ab57-9c8d29a0e004\") " pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:33:18.326990 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.326951    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f565450e-c855-4701-ab57-9c8d29a0e004-tmp-dir\") pod \"explicit-model-808e44-kserve-69f85f47b7-crp2h\" (UID: \"f565450e-c855-4701-ab57-9c8d29a0e004\") " pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:33:18.427933 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.427868    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f565450e-c855-4701-ab57-9c8d29a0e004-tls-certs\") pod \"explicit-model-808e44-kserve-69f85f47b7-crp2h\" (UID: \"f565450e-c855-4701-ab57-9c8d29a0e004\") " pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:33:18.428133 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.427976    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql968\" (UniqueName: \"kubernetes.io/projected/f565450e-c855-4701-ab57-9c8d29a0e004-kube-api-access-ql968\") pod \"explicit-model-808e44-kserve-69f85f47b7-crp2h\" (UID: \"f565450e-c855-4701-ab57-9c8d29a0e004\") " pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:33:18.428133 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.428064    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f565450e-c855-4701-ab57-9c8d29a0e004-home\") pod \"explicit-model-808e44-kserve-69f85f47b7-crp2h\" (UID: \"f565450e-c855-4701-ab57-9c8d29a0e004\") " pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:33:18.428133 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.428092    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f565450e-c855-4701-ab57-9c8d29a0e004-dshm\") pod \"explicit-model-808e44-kserve-69f85f47b7-crp2h\" (UID: \"f565450e-c855-4701-ab57-9c8d29a0e004\") " pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:33:18.428133 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.428112    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f565450e-c855-4701-ab57-9c8d29a0e004-model-cache\") pod \"explicit-model-808e44-kserve-69f85f47b7-crp2h\" (UID: \"f565450e-c855-4701-ab57-9c8d29a0e004\") " pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:33:18.428321 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.428138    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f565450e-c855-4701-ab57-9c8d29a0e004-tmp-dir\") pod \"explicit-model-808e44-kserve-69f85f47b7-crp2h\" (UID: \"f565450e-c855-4701-ab57-9c8d29a0e004\") " pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:33:18.428440 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.428418    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f565450e-c855-4701-ab57-9c8d29a0e004-home\") pod \"explicit-model-808e44-kserve-69f85f47b7-crp2h\" (UID: \"f565450e-c855-4701-ab57-9c8d29a0e004\") " pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:33:18.428513 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.428500    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f565450e-c855-4701-ab57-9c8d29a0e004-model-cache\") pod \"explicit-model-808e44-kserve-69f85f47b7-crp2h\" (UID: \"f565450e-c855-4701-ab57-9c8d29a0e004\") " pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:33:18.428592 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.428572    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f565450e-c855-4701-ab57-9c8d29a0e004-tmp-dir\") pod \"explicit-model-808e44-kserve-69f85f47b7-crp2h\" (UID: \"f565450e-c855-4701-ab57-9c8d29a0e004\") " pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:33:18.430451 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.430423    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f565450e-c855-4701-ab57-9c8d29a0e004-dshm\") pod \"explicit-model-808e44-kserve-69f85f47b7-crp2h\" (UID: \"f565450e-c855-4701-ab57-9c8d29a0e004\") " pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:33:18.430845 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.430827    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f565450e-c855-4701-ab57-9c8d29a0e004-tls-certs\") pod \"explicit-model-808e44-kserve-69f85f47b7-crp2h\" (UID: \"f565450e-c855-4701-ab57-9c8d29a0e004\") " pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:33:18.436528 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.436509    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql968\" (UniqueName: \"kubernetes.io/projected/f565450e-c855-4701-ab57-9c8d29a0e004-kube-api-access-ql968\") pod \"explicit-model-808e44-kserve-69f85f47b7-crp2h\" (UID: \"f565450e-c855-4701-ab57-9c8d29a0e004\") " pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:33:18.584196 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.584109    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:33:18.860869 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:18.860841    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"]
Sep 18 13:33:18.863768 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:33:18.863738    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf565450e_c855_4701_ab57_9c8d29a0e004.slice/crio-18c3aedcf4a70e90599af9867e61c6b17f0f381d2b6c2bb91b91891141edec8f WatchSource:0}: Error finding container 18c3aedcf4a70e90599af9867e61c6b17f0f381d2b6c2bb91b91891141edec8f: Status 404 returned error can't find the container with id 18c3aedcf4a70e90599af9867e61c6b17f0f381d2b6c2bb91b91891141edec8f
Sep 18 13:33:19.791294 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:19.791254    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h" event={"ID":"f565450e-c855-4701-ab57-9c8d29a0e004","Type":"ContainerStarted","Data":"0ced90375a7ba6901f9e3f0341f3747c5437ff727b5ef509341a6bb953d0d842"}
Sep 18 13:33:19.791665 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:19.791302    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h" event={"ID":"f565450e-c855-4701-ab57-9c8d29a0e004","Type":"ContainerStarted","Data":"18c3aedcf4a70e90599af9867e61c6b17f0f381d2b6c2bb91b91891141edec8f"}
Sep 18 13:33:19.815484 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:19.815431    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h" podStartSLOduration=1.815415502 podStartE2EDuration="1.815415502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:33:19.812810159 +0000 UTC m=+1097.168269985" watchObservedRunningTime="2026-09-18 13:33:19.815415502 +0000 UTC m=+1097.170875330"
Sep 18 13:33:24.997952 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:24.997917    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:33:28.584585 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:28.584545    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:33:28.584585 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:28.584595    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:33:28.597090 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:28.597066    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:33:28.850698 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:28.850620    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:33:45.458729 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:45.458689    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw"]
Sep 18 13:33:45.462646 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:45.462630    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw"
Sep 18 13:33:45.465813 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:45.465788    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-bb3ce27c\""
Sep 18 13:33:45.465981 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:45.465960    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-bb3ce27c-dockercfg-mfsq4\""
Sep 18 13:33:45.477415 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:45.477390    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw"]
Sep 18 13:33:45.598037 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:45.597995    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbqcf\" (UniqueName: \"kubernetes.io/projected/75405001-428f-44ee-942b-a7042c97c05f-kube-api-access-fbqcf\") pod \"payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw\" (UID: \"75405001-428f-44ee-942b-a7042c97c05f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw"
Sep 18 13:33:45.598209 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:45.598126    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75405001-428f-44ee-942b-a7042c97c05f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw\" (UID: \"75405001-428f-44ee-942b-a7042c97c05f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw"
Sep 18 13:33:45.699148 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:45.699101    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbqcf\" (UniqueName: \"kubernetes.io/projected/75405001-428f-44ee-942b-a7042c97c05f-kube-api-access-fbqcf\") pod \"payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw\" (UID: \"75405001-428f-44ee-942b-a7042c97c05f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw"
Sep 18 13:33:45.699351 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:45.699230    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75405001-428f-44ee-942b-a7042c97c05f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw\" (UID: \"75405001-428f-44ee-942b-a7042c97c05f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw"
Sep 18 13:33:45.700007 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:45.699988    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75405001-428f-44ee-942b-a7042c97c05f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw\" (UID: \"75405001-428f-44ee-942b-a7042c97c05f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw"
Sep 18 13:33:45.707497 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:45.707463    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbqcf\" (UniqueName: \"kubernetes.io/projected/75405001-428f-44ee-942b-a7042c97c05f-kube-api-access-fbqcf\") pod \"payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw\" (UID: \"75405001-428f-44ee-942b-a7042c97c05f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw"
Sep 18 13:33:45.773712 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:45.773607    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw"
Sep 18 13:33:46.071251 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:46.071220    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw"]
Sep 18 13:33:46.072329 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:33:46.072288    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75405001_428f_44ee_942b_a7042c97c05f.slice/crio-ad058643686f0f4e6e5850b0d251a7b5e005d3fa3d19ab9f0c82b932d589620c WatchSource:0}: Error finding container ad058643686f0f4e6e5850b0d251a7b5e005d3fa3d19ab9f0c82b932d589620c: Status 404 returned error can't find the container with id ad058643686f0f4e6e5850b0d251a7b5e005d3fa3d19ab9f0c82b932d589620c
Sep 18 13:33:46.927142 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:46.927102    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw" event={"ID":"75405001-428f-44ee-942b-a7042c97c05f","Type":"ContainerStarted","Data":"4a7c1d981ec8a981f8cef53e9760b4b0b69f21b933f0acdc7b7191b907a1092c"}
Sep 18 13:33:46.927563 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:46.927163    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw" event={"ID":"75405001-428f-44ee-942b-a7042c97c05f","Type":"ContainerStarted","Data":"ad058643686f0f4e6e5850b0d251a7b5e005d3fa3d19ab9f0c82b932d589620c"}
Sep 18 13:33:46.927563 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:46.927201    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw"
Sep 18 13:33:46.946749 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:46.946700    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw" podStartSLOduration=1.946685862 podStartE2EDuration="1.946685862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:33:46.944628491 +0000 UTC m=+1124.300088316" watchObservedRunningTime="2026-09-18 13:33:46.946685862 +0000 UTC m=+1124.302145696"
Sep 18 13:33:50.335496 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:50.335462    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-808e44-cgd5n"]
Sep 18 13:33:50.338730 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:50.338711    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-808e44-cgd5n"
Sep 18 13:33:50.341234 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:50.341212    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nh2cd\""
Sep 18 13:33:50.341577 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:50.341562    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:33:50.347084 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:50.347058    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4116519-d4ba-41ea-b022-978de6cdc847-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-808e44-cgd5n\" (UID: \"b4116519-d4ba-41ea-b022-978de6cdc847\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-808e44-cgd5n"
Sep 18 13:33:50.358832 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:50.358814    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-808e44-cgd5n"]
Sep 18 13:33:50.447556 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:50.447524    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4116519-d4ba-41ea-b022-978de6cdc847-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-808e44-cgd5n\" (UID: \"b4116519-d4ba-41ea-b022-978de6cdc847\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-808e44-cgd5n"
Sep 18 13:33:50.448207 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:50.448188    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4116519-d4ba-41ea-b022-978de6cdc847-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-808e44-cgd5n\" (UID: \"b4116519-d4ba-41ea-b022-978de6cdc847\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-808e44-cgd5n"
Sep 18 13:33:50.648820 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:50.648792    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-808e44-cgd5n"
Sep 18 13:33:50.928927 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:50.928868    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-808e44-cgd5n"]
Sep 18 13:33:50.947122 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:50.947092    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-808e44-cgd5n" event={"ID":"b4116519-d4ba-41ea-b022-978de6cdc847","Type":"ContainerStarted","Data":"7bd3afa8e5ae1492d36655a471fa1c712b9a4565ecb1d6a985de9adb19cca30d"}
Sep 18 13:33:51.953973 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:51.953936    2586 generic.go:356] "Generic (PLEG): container finished" podID="b4116519-d4ba-41ea-b022-978de6cdc847" containerID="e0a90d7335aef6213befa4f4206649ee18441226e42fdab2a7647f6acbf030da" exitCode=0
Sep 18 13:33:51.954360 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:51.954020    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-808e44-cgd5n" event={"ID":"b4116519-d4ba-41ea-b022-978de6cdc847","Type":"ContainerDied","Data":"e0a90d7335aef6213befa4f4206649ee18441226e42fdab2a7647f6acbf030da"}
Sep 18 13:33:53.094389 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:53.094366    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-808e44-cgd5n"
Sep 18 13:33:53.171087 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:53.171054    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4116519-d4ba-41ea-b022-978de6cdc847-maas-api-service-ca\") pod \"b4116519-d4ba-41ea-b022-978de6cdc847\" (UID: \"b4116519-d4ba-41ea-b022-978de6cdc847\") "
Sep 18 13:33:53.171429 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:53.171407    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4116519-d4ba-41ea-b022-978de6cdc847-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b4116519-d4ba-41ea-b022-978de6cdc847" (UID: "b4116519-d4ba-41ea-b022-978de6cdc847"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:33:53.272267 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:53.272178    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4116519-d4ba-41ea-b022-978de6cdc847-maas-api-service-ca\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:33:53.965305 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:53.965266    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-808e44-cgd5n" event={"ID":"b4116519-d4ba-41ea-b022-978de6cdc847","Type":"ContainerDied","Data":"7bd3afa8e5ae1492d36655a471fa1c712b9a4565ecb1d6a985de9adb19cca30d"}
Sep 18 13:33:53.965305 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:53.965300    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-808e44-cgd5n"
Sep 18 13:33:53.965525 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:53.965328    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bd3afa8e5ae1492d36655a471fa1c712b9a4565ecb1d6a985de9adb19cca30d"
Sep 18 13:33:55.107681 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:55.107643    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:33:57.934430 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:33:57.934396    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw"
Sep 18 13:34:03.315270 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:03.315234    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-808e44-cgd5n"]
Sep 18 13:34:03.317414 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:03.317390    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-808e44-cgd5n"]
Sep 18 13:34:04.850298 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:04.850257    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4116519-d4ba-41ea-b022-978de6cdc847" path="/var/lib/kubelet/pods/b4116519-d4ba-41ea-b022-978de6cdc847/volumes"
Sep 18 13:34:08.666411 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:08.666379    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"]
Sep 18 13:34:22.357145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:22.357055    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw"]
Sep 18 13:34:22.357731 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:22.357330    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw" podUID="75405001-428f-44ee-942b-a7042c97c05f" containerName="payload-pre-processing" containerID="cri-o://4a7c1d981ec8a981f8cef53e9760b4b0b69f21b933f0acdc7b7191b907a1092c" gracePeriod=30
Sep 18 13:34:22.618609 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:22.618543    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw"
Sep 18 13:34:22.770644 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:22.770608    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75405001-428f-44ee-942b-a7042c97c05f-plugins-config-volume\") pod \"75405001-428f-44ee-942b-a7042c97c05f\" (UID: \"75405001-428f-44ee-942b-a7042c97c05f\") "
Sep 18 13:34:22.770810 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:22.770684    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbqcf\" (UniqueName: \"kubernetes.io/projected/75405001-428f-44ee-942b-a7042c97c05f-kube-api-access-fbqcf\") pod \"75405001-428f-44ee-942b-a7042c97c05f\" (UID: \"75405001-428f-44ee-942b-a7042c97c05f\") "
Sep 18 13:34:22.770986 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:22.770965    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75405001-428f-44ee-942b-a7042c97c05f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "75405001-428f-44ee-942b-a7042c97c05f" (UID: "75405001-428f-44ee-942b-a7042c97c05f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:34:22.772842 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:22.772823    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75405001-428f-44ee-942b-a7042c97c05f-kube-api-access-fbqcf" (OuterVolumeSpecName: "kube-api-access-fbqcf") pod "75405001-428f-44ee-942b-a7042c97c05f" (UID: "75405001-428f-44ee-942b-a7042c97c05f"). InnerVolumeSpecName "kube-api-access-fbqcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:34:22.872262 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:22.872203    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbqcf\" (UniqueName: \"kubernetes.io/projected/75405001-428f-44ee-942b-a7042c97c05f-kube-api-access-fbqcf\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:34:22.872262 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:22.872226    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75405001-428f-44ee-942b-a7042c97c05f-plugins-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:34:23.125032 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:23.124953    2586 generic.go:356] "Generic (PLEG): container finished" podID="75405001-428f-44ee-942b-a7042c97c05f" containerID="4a7c1d981ec8a981f8cef53e9760b4b0b69f21b933f0acdc7b7191b907a1092c" exitCode=0
Sep 18 13:34:23.125032 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:23.124992    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw"
Sep 18 13:34:23.125212 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:23.125035    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw" event={"ID":"75405001-428f-44ee-942b-a7042c97c05f","Type":"ContainerDied","Data":"4a7c1d981ec8a981f8cef53e9760b4b0b69f21b933f0acdc7b7191b907a1092c"}
Sep 18 13:34:23.125212 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:23.125082    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw" event={"ID":"75405001-428f-44ee-942b-a7042c97c05f","Type":"ContainerDied","Data":"ad058643686f0f4e6e5850b0d251a7b5e005d3fa3d19ab9f0c82b932d589620c"}
Sep 18 13:34:23.125212 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:23.125103    2586 scope.go:117] "RemoveContainer" containerID="4a7c1d981ec8a981f8cef53e9760b4b0b69f21b933f0acdc7b7191b907a1092c"
Sep 18 13:34:23.136672 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:23.136651    2586 scope.go:117] "RemoveContainer" containerID="4a7c1d981ec8a981f8cef53e9760b4b0b69f21b933f0acdc7b7191b907a1092c"
Sep 18 13:34:23.136946 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:34:23.136928    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a7c1d981ec8a981f8cef53e9760b4b0b69f21b933f0acdc7b7191b907a1092c\": container with ID starting with 4a7c1d981ec8a981f8cef53e9760b4b0b69f21b933f0acdc7b7191b907a1092c not found: ID does not exist" containerID="4a7c1d981ec8a981f8cef53e9760b4b0b69f21b933f0acdc7b7191b907a1092c"
Sep 18 13:34:23.136999 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:23.136954    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7c1d981ec8a981f8cef53e9760b4b0b69f21b933f0acdc7b7191b907a1092c"} err="failed to get container status \"4a7c1d981ec8a981f8cef53e9760b4b0b69f21b933f0acdc7b7191b907a1092c\": rpc error: code = NotFound desc = could not find container \"4a7c1d981ec8a981f8cef53e9760b4b0b69f21b933f0acdc7b7191b907a1092c\": container with ID starting with 4a7c1d981ec8a981f8cef53e9760b4b0b69f21b933f0acdc7b7191b907a1092c not found: ID does not exist"
Sep 18 13:34:23.153738 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:23.153714    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw"]
Sep 18 13:34:23.162478 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:23.162458    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-bb3ce27c-bd84c87db-6j4bw"]
Sep 18 13:34:23.686423 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:23.686351    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h" podUID="f565450e-c855-4701-ab57-9c8d29a0e004" containerName="main" containerID="cri-o://0ced90375a7ba6901f9e3f0341f3747c5437ff727b5ef509341a6bb953d0d842" gracePeriod=45
Sep 18 13:34:23.935802 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:23.935779    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:34:24.084700 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.084629    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f565450e-c855-4701-ab57-9c8d29a0e004-tmp-dir\") pod \"f565450e-c855-4701-ab57-9c8d29a0e004\" (UID: \"f565450e-c855-4701-ab57-9c8d29a0e004\") "
Sep 18 13:34:24.084700 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.084662    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f565450e-c855-4701-ab57-9c8d29a0e004-home\") pod \"f565450e-c855-4701-ab57-9c8d29a0e004\" (UID: \"f565450e-c855-4701-ab57-9c8d29a0e004\") "
Sep 18 13:34:24.084926 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.084708    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f565450e-c855-4701-ab57-9c8d29a0e004-tls-certs\") pod \"f565450e-c855-4701-ab57-9c8d29a0e004\" (UID: \"f565450e-c855-4701-ab57-9c8d29a0e004\") "
Sep 18 13:34:24.084926 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.084755    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f565450e-c855-4701-ab57-9c8d29a0e004-dshm\") pod \"f565450e-c855-4701-ab57-9c8d29a0e004\" (UID: \"f565450e-c855-4701-ab57-9c8d29a0e004\") "
Sep 18 13:34:24.084926 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.084776    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f565450e-c855-4701-ab57-9c8d29a0e004-model-cache\") pod \"f565450e-c855-4701-ab57-9c8d29a0e004\" (UID: \"f565450e-c855-4701-ab57-9c8d29a0e004\") "
Sep 18 13:34:24.084926 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.084802    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ql968\" (UniqueName: \"kubernetes.io/projected/f565450e-c855-4701-ab57-9c8d29a0e004-kube-api-access-ql968\") pod \"f565450e-c855-4701-ab57-9c8d29a0e004\" (UID: \"f565450e-c855-4701-ab57-9c8d29a0e004\") "
Sep 18 13:34:24.085145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.084949    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f565450e-c855-4701-ab57-9c8d29a0e004-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f565450e-c855-4701-ab57-9c8d29a0e004" (UID: "f565450e-c855-4701-ab57-9c8d29a0e004"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:34:24.085145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.084982    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f565450e-c855-4701-ab57-9c8d29a0e004-home" (OuterVolumeSpecName: "home") pod "f565450e-c855-4701-ab57-9c8d29a0e004" (UID: "f565450e-c855-4701-ab57-9c8d29a0e004"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:34:24.085145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.085095    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f565450e-c855-4701-ab57-9c8d29a0e004-model-cache" (OuterVolumeSpecName: "model-cache") pod "f565450e-c855-4701-ab57-9c8d29a0e004" (UID: "f565450e-c855-4701-ab57-9c8d29a0e004"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:34:24.085145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.085137    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f565450e-c855-4701-ab57-9c8d29a0e004-tmp-dir\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:34:24.085289 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.085157    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f565450e-c855-4701-ab57-9c8d29a0e004-home\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:34:24.087027 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.086996    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f565450e-c855-4701-ab57-9c8d29a0e004-kube-api-access-ql968" (OuterVolumeSpecName: "kube-api-access-ql968") pod "f565450e-c855-4701-ab57-9c8d29a0e004" (UID: "f565450e-c855-4701-ab57-9c8d29a0e004"). InnerVolumeSpecName "kube-api-access-ql968". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:34:24.087143 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.087039    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f565450e-c855-4701-ab57-9c8d29a0e004-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f565450e-c855-4701-ab57-9c8d29a0e004" (UID: "f565450e-c855-4701-ab57-9c8d29a0e004"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:34:24.087143 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.087077    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f565450e-c855-4701-ab57-9c8d29a0e004-dshm" (OuterVolumeSpecName: "dshm") pod "f565450e-c855-4701-ab57-9c8d29a0e004" (UID: "f565450e-c855-4701-ab57-9c8d29a0e004"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:34:24.132218 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.132188    2586 generic.go:356] "Generic (PLEG): container finished" podID="f565450e-c855-4701-ab57-9c8d29a0e004" containerID="0ced90375a7ba6901f9e3f0341f3747c5437ff727b5ef509341a6bb953d0d842" exitCode=0
Sep 18 13:34:24.132358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.132238    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"
Sep 18 13:34:24.132358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.132278    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h" event={"ID":"f565450e-c855-4701-ab57-9c8d29a0e004","Type":"ContainerDied","Data":"0ced90375a7ba6901f9e3f0341f3747c5437ff727b5ef509341a6bb953d0d842"}
Sep 18 13:34:24.132358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.132323    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h" event={"ID":"f565450e-c855-4701-ab57-9c8d29a0e004","Type":"ContainerDied","Data":"18c3aedcf4a70e90599af9867e61c6b17f0f381d2b6c2bb91b91891141edec8f"}
Sep 18 13:34:24.132358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.132343    2586 scope.go:117] "RemoveContainer" containerID="0ced90375a7ba6901f9e3f0341f3747c5437ff727b5ef509341a6bb953d0d842"
Sep 18 13:34:24.141656 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.141626    2586 scope.go:117] "RemoveContainer" containerID="0ced90375a7ba6901f9e3f0341f3747c5437ff727b5ef509341a6bb953d0d842"
Sep 18 13:34:24.141993 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:34:24.141955    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ced90375a7ba6901f9e3f0341f3747c5437ff727b5ef509341a6bb953d0d842\": container with ID starting with 0ced90375a7ba6901f9e3f0341f3747c5437ff727b5ef509341a6bb953d0d842 not found: ID does not exist" containerID="0ced90375a7ba6901f9e3f0341f3747c5437ff727b5ef509341a6bb953d0d842"
Sep 18 13:34:24.142093 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.142010    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ced90375a7ba6901f9e3f0341f3747c5437ff727b5ef509341a6bb953d0d842"} err="failed to get container status \"0ced90375a7ba6901f9e3f0341f3747c5437ff727b5ef509341a6bb953d0d842\": rpc error: code = NotFound desc = could not find container \"0ced90375a7ba6901f9e3f0341f3747c5437ff727b5ef509341a6bb953d0d842\": container with ID starting with 0ced90375a7ba6901f9e3f0341f3747c5437ff727b5ef509341a6bb953d0d842 not found: ID does not exist"
Sep 18 13:34:24.168916 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.168878    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"]
Sep 18 13:34:24.179934 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.179914    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-808e44/explicit-model-808e44-kserve-69f85f47b7-crp2h"]
Sep 18 13:34:24.186579 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.186559    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f565450e-c855-4701-ab57-9c8d29a0e004-tls-certs\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:34:24.186651 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.186581    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f565450e-c855-4701-ab57-9c8d29a0e004-dshm\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:34:24.186651 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.186590    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f565450e-c855-4701-ab57-9c8d29a0e004-model-cache\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:34:24.186651 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.186598    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql968\" (UniqueName: \"kubernetes.io/projected/f565450e-c855-4701-ab57-9c8d29a0e004-kube-api-access-ql968\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:34:24.850250 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.850218    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75405001-428f-44ee-942b-a7042c97c05f" path="/var/lib/kubelet/pods/75405001-428f-44ee-942b-a7042c97c05f/volumes"
Sep 18 13:34:24.850644 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:24.850631    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f565450e-c855-4701-ab57-9c8d29a0e004" path="/var/lib/kubelet/pods/f565450e-c855-4701-ab57-9c8d29a0e004/volumes"
Sep 18 13:34:26.910207 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:26.910169    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:34:48.190557 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:48.190519    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"]
Sep 18 13:34:48.190977 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:48.190785    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg" podUID="2a0f684d-5cab-42d6-84b1-20cc2470808b" containerName="istio-proxy" containerID="cri-o://cf62afcb3f80e1dd6c726fec80e086607c5b3a001493b5285e789848bda9d447" gracePeriod=30
Sep 18 13:34:48.993984 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:48.993946    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"]
Sep 18 13:34:48.995118 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:48.995097    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75405001-428f-44ee-942b-a7042c97c05f" containerName="payload-pre-processing"
Sep 18 13:34:48.995246 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:48.995236    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="75405001-428f-44ee-942b-a7042c97c05f" containerName="payload-pre-processing"
Sep 18 13:34:48.995376 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:48.995367    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f565450e-c855-4701-ab57-9c8d29a0e004" containerName="main"
Sep 18 13:34:48.995452 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:48.995443    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f565450e-c855-4701-ab57-9c8d29a0e004" containerName="main"
Sep 18 13:34:48.995546 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:48.995537    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4116519-d4ba-41ea-b022-978de6cdc847" containerName="revoke-keys"
Sep 18 13:34:48.995619 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:48.995611    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4116519-d4ba-41ea-b022-978de6cdc847" containerName="revoke-keys"
Sep 18 13:34:48.995879 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:48.995868    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="75405001-428f-44ee-942b-a7042c97c05f" containerName="payload-pre-processing"
Sep 18 13:34:48.995997 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:48.995986    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4116519-d4ba-41ea-b022-978de6cdc847" containerName="revoke-keys"
Sep 18 13:34:48.996082 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:48.996073    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f565450e-c855-4701-ab57-9c8d29a0e004" containerName="main"
Sep 18 13:34:49.004377 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.004351    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.009246 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.009226    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-b81e47-openshift-default-dockercfg-c5697\""
Sep 18 13:34:49.026734 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.026710    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"]
Sep 18 13:34:49.131993 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.131954    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istio-podinfo\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.132157 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.132013    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istio-token\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.132157 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.132085    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-workload-socket\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.132290 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.132163    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istio-data\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.132290 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.132196    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-credential-socket\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.132290 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.132236    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-workload-certs\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.132290 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.132258    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istiod-ca-cert\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.132290 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.132287    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istio-envoy\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.132489 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.132366    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq24r\" (UniqueName: \"kubernetes.io/projected/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-kube-api-access-wq24r\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.233145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.233113    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istio-podinfo\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.233625 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.233157    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istio-token\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.233625 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.233295    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-workload-socket\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.233625 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.233352    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istio-data\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.233625 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.233400    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-credential-socket\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.233625 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.233447    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-workload-certs\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.233625 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.233492    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istiod-ca-cert\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.233625 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.233521    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istio-envoy\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.233625 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.233563    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq24r\" (UniqueName: \"kubernetes.io/projected/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-kube-api-access-wq24r\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.234081 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.233719    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istio-data\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.234081 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.233750    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-workload-socket\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.234081 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.233777    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-credential-socket\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.234081 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.233879    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-workload-certs\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.234590 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.234499    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istiod-ca-cert\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.235941 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.235921    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istio-podinfo\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.236103 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.236081    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istio-envoy\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.246571 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.246506    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq24r\" (UniqueName: \"kubernetes.io/projected/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-kube-api-access-wq24r\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.246662 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.246635    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istio-token\") pod \"e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") " pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.315419 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.315388    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:49.650637 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.650611    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"]
Sep 18 13:34:49.653445 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:34:49.653415    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a939872_c16d_4d63_b20e_7f2ae7dc28cf.slice/crio-433ef2fb56ffaac61e10affc3cd55eca5d0b62a872ea8d91b4fed606a8e2d4f3 WatchSource:0}: Error finding container 433ef2fb56ffaac61e10affc3cd55eca5d0b62a872ea8d91b4fed606a8e2d4f3: Status 404 returned error can't find the container with id 433ef2fb56ffaac61e10affc3cd55eca5d0b62a872ea8d91b4fed606a8e2d4f3
Sep 18 13:34:49.655723 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.655689    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:34:49.655795 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.655761    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:34:49.655795 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:49.655791    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:34:50.268353 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:50.268318    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg" event={"ID":"1a939872-c16d-4d63-b20e-7f2ae7dc28cf","Type":"ContainerStarted","Data":"0c626262110dcd2a249ddfb03742cd8f5e70ddfffe041b52924f93003f142f20"}
Sep 18 13:34:50.268793 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:50.268374    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg" event={"ID":"1a939872-c16d-4d63-b20e-7f2ae7dc28cf","Type":"ContainerStarted","Data":"433ef2fb56ffaac61e10affc3cd55eca5d0b62a872ea8d91b4fed606a8e2d4f3"}
Sep 18 13:34:50.315681 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:50.315644    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:50.321325 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:50.321299    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:50.321621 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:50.321574    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg" podStartSLOduration=2.321557595 podStartE2EDuration="2.321557595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:34:50.315945819 +0000 UTC m=+1187.671405644" watchObservedRunningTime="2026-09-18 13:34:50.321557595 +0000 UTC m=+1187.677017420"
Sep 18 13:34:51.273455 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:51.273421    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:51.274324 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:51.274304    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:34:53.435270 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.435245    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:34:53.578627 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.578529    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-istio-data\") pod \"2a0f684d-5cab-42d6-84b1-20cc2470808b\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") "
Sep 18 13:34:53.578627 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.578573    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-istio-envoy\") pod \"2a0f684d-5cab-42d6-84b1-20cc2470808b\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") "
Sep 18 13:34:53.578627 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.578625    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-workload-certs\") pod \"2a0f684d-5cab-42d6-84b1-20cc2470808b\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") "
Sep 18 13:34:53.578946 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.578655    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a0f684d-5cab-42d6-84b1-20cc2470808b-istio-podinfo\") pod \"2a0f684d-5cab-42d6-84b1-20cc2470808b\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") "
Sep 18 13:34:53.578946 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.578710    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a0f684d-5cab-42d6-84b1-20cc2470808b-istiod-ca-cert\") pod \"2a0f684d-5cab-42d6-84b1-20cc2470808b\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") "
Sep 18 13:34:53.578946 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.578738    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-credential-socket\") pod \"2a0f684d-5cab-42d6-84b1-20cc2470808b\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") "
Sep 18 13:34:53.578946 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.578782    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-workload-socket\") pod \"2a0f684d-5cab-42d6-84b1-20cc2470808b\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") "
Sep 18 13:34:53.578946 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.578812    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5c5x\" (UniqueName: \"kubernetes.io/projected/2a0f684d-5cab-42d6-84b1-20cc2470808b-kube-api-access-f5c5x\") pod \"2a0f684d-5cab-42d6-84b1-20cc2470808b\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") "
Sep 18 13:34:53.578946 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.578835    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-istio-data" (OuterVolumeSpecName: "istio-data") pod "2a0f684d-5cab-42d6-84b1-20cc2470808b" (UID: "2a0f684d-5cab-42d6-84b1-20cc2470808b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:34:53.578946 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.578850    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a0f684d-5cab-42d6-84b1-20cc2470808b-istio-token\") pod \"2a0f684d-5cab-42d6-84b1-20cc2470808b\" (UID: \"2a0f684d-5cab-42d6-84b1-20cc2470808b\") "
Sep 18 13:34:53.579310 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.578980    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2a0f684d-5cab-42d6-84b1-20cc2470808b" (UID: "2a0f684d-5cab-42d6-84b1-20cc2470808b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:34:53.579310 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.579051    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2a0f684d-5cab-42d6-84b1-20cc2470808b" (UID: "2a0f684d-5cab-42d6-84b1-20cc2470808b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:34:53.579310 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.579137    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2a0f684d-5cab-42d6-84b1-20cc2470808b" (UID: "2a0f684d-5cab-42d6-84b1-20cc2470808b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:34:53.579310 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.579183    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0f684d-5cab-42d6-84b1-20cc2470808b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2a0f684d-5cab-42d6-84b1-20cc2470808b" (UID: "2a0f684d-5cab-42d6-84b1-20cc2470808b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:34:53.579484 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.579315    2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-istio-data\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:34:53.579484 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.579334    2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-workload-certs\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:34:53.579484 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.579348    2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a0f684d-5cab-42d6-84b1-20cc2470808b-istiod-ca-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:34:53.579484 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.579361    2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-credential-socket\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:34:53.579484 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.579376    2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-workload-socket\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:34:53.581677 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.581657    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2a0f684d-5cab-42d6-84b1-20cc2470808b" (UID: "2a0f684d-5cab-42d6-84b1-20cc2470808b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:34:53.581677 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.581659    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a0f684d-5cab-42d6-84b1-20cc2470808b-kube-api-access-f5c5x" (OuterVolumeSpecName: "kube-api-access-f5c5x") pod "2a0f684d-5cab-42d6-84b1-20cc2470808b" (UID: "2a0f684d-5cab-42d6-84b1-20cc2470808b"). InnerVolumeSpecName "kube-api-access-f5c5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:34:53.581790 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.581698    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a0f684d-5cab-42d6-84b1-20cc2470808b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2a0f684d-5cab-42d6-84b1-20cc2470808b" (UID: "2a0f684d-5cab-42d6-84b1-20cc2470808b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 13:34:53.581790 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.581703    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a0f684d-5cab-42d6-84b1-20cc2470808b-istio-token" (OuterVolumeSpecName: "istio-token") pod "2a0f684d-5cab-42d6-84b1-20cc2470808b" (UID: "2a0f684d-5cab-42d6-84b1-20cc2470808b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:34:53.680296 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.680263    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5c5x\" (UniqueName: \"kubernetes.io/projected/2a0f684d-5cab-42d6-84b1-20cc2470808b-kube-api-access-f5c5x\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:34:53.680296 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.680293    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a0f684d-5cab-42d6-84b1-20cc2470808b-istio-token\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:34:53.680296 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.680302    2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a0f684d-5cab-42d6-84b1-20cc2470808b-istio-envoy\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:34:53.680518 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:53.680310    2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a0f684d-5cab-42d6-84b1-20cc2470808b-istio-podinfo\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:34:54.290435 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:54.290403    2586 generic.go:356] "Generic (PLEG): container finished" podID="2a0f684d-5cab-42d6-84b1-20cc2470808b" containerID="cf62afcb3f80e1dd6c726fec80e086607c5b3a001493b5285e789848bda9d447" exitCode=0
Sep 18 13:34:54.290613 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:54.290448    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"
Sep 18 13:34:54.290613 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:54.290498    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg" event={"ID":"2a0f684d-5cab-42d6-84b1-20cc2470808b","Type":"ContainerDied","Data":"cf62afcb3f80e1dd6c726fec80e086607c5b3a001493b5285e789848bda9d447"}
Sep 18 13:34:54.290613 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:54.290561    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg" event={"ID":"2a0f684d-5cab-42d6-84b1-20cc2470808b","Type":"ContainerDied","Data":"5dd276b842c0699c551ef0fef33fb1a5d983f4ab7abc491d70cc9628f3776735"}
Sep 18 13:34:54.290613 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:54.290597    2586 scope.go:117] "RemoveContainer" containerID="cf62afcb3f80e1dd6c726fec80e086607c5b3a001493b5285e789848bda9d447"
Sep 18 13:34:54.300353 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:54.300337    2586 scope.go:117] "RemoveContainer" containerID="ac1e77a63be3096441355789528322fb4d634a121d9cd3fe3fbde651cdd1bf9a"
Sep 18 13:34:54.310105 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:54.309890    2586 scope.go:117] "RemoveContainer" containerID="cf62afcb3f80e1dd6c726fec80e086607c5b3a001493b5285e789848bda9d447"
Sep 18 13:34:54.310228 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:34:54.310207    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf62afcb3f80e1dd6c726fec80e086607c5b3a001493b5285e789848bda9d447\": container with ID starting with cf62afcb3f80e1dd6c726fec80e086607c5b3a001493b5285e789848bda9d447 not found: ID does not exist" containerID="cf62afcb3f80e1dd6c726fec80e086607c5b3a001493b5285e789848bda9d447"
Sep 18 13:34:54.310317 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:54.310236    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf62afcb3f80e1dd6c726fec80e086607c5b3a001493b5285e789848bda9d447"} err="failed to get container status \"cf62afcb3f80e1dd6c726fec80e086607c5b3a001493b5285e789848bda9d447\": rpc error: code = NotFound desc = could not find container \"cf62afcb3f80e1dd6c726fec80e086607c5b3a001493b5285e789848bda9d447\": container with ID starting with cf62afcb3f80e1dd6c726fec80e086607c5b3a001493b5285e789848bda9d447 not found: ID does not exist"
Sep 18 13:34:54.310317 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:54.310253    2586 scope.go:117] "RemoveContainer" containerID="ac1e77a63be3096441355789528322fb4d634a121d9cd3fe3fbde651cdd1bf9a"
Sep 18 13:34:54.310494 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:34:54.310478    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1e77a63be3096441355789528322fb4d634a121d9cd3fe3fbde651cdd1bf9a\": container with ID starting with ac1e77a63be3096441355789528322fb4d634a121d9cd3fe3fbde651cdd1bf9a not found: ID does not exist" containerID="ac1e77a63be3096441355789528322fb4d634a121d9cd3fe3fbde651cdd1bf9a"
Sep 18 13:34:54.310535 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:54.310499    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1e77a63be3096441355789528322fb4d634a121d9cd3fe3fbde651cdd1bf9a"} err="failed to get container status \"ac1e77a63be3096441355789528322fb4d634a121d9cd3fe3fbde651cdd1bf9a\": rpc error: code = NotFound desc = could not find container \"ac1e77a63be3096441355789528322fb4d634a121d9cd3fe3fbde651cdd1bf9a\": container with ID starting with ac1e77a63be3096441355789528322fb4d634a121d9cd3fe3fbde651cdd1bf9a not found: ID does not exist"
Sep 18 13:34:54.343650 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:54.343622    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"]
Sep 18 13:34:54.371735 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:54.371707    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-808e44-openshift-default-699c89ff44-vlgbg"]
Sep 18 13:34:54.850049 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:54.850018    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a0f684d-5cab-42d6-84b1-20cc2470808b" path="/var/lib/kubelet/pods/2a0f684d-5cab-42d6-84b1-20cc2470808b/volumes"
Sep 18 13:34:58.264928 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.264873    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl"]
Sep 18 13:34:58.265299 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.265255    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a0f684d-5cab-42d6-84b1-20cc2470808b" containerName="istio-proxy"
Sep 18 13:34:58.265299 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.265266    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a0f684d-5cab-42d6-84b1-20cc2470808b" containerName="istio-proxy"
Sep 18 13:34:58.265379 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.265342    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a0f684d-5cab-42d6-84b1-20cc2470808b" containerName="istio-proxy"
Sep 18 13:34:58.265379 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.265351    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a0f684d-5cab-42d6-84b1-20cc2470808b" containerName="istio-proxy"
Sep 18 13:34:58.269863 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.269845    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl"
Sep 18 13:34:58.280494 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.280468    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-b2d5da-dockercfg-znb8f\""
Sep 18 13:34:58.280621 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.280510    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-b2d5da\""
Sep 18 13:34:58.326742 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.326711    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl"]
Sep 18 13:34:58.358418 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.358386    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl"]
Sep 18 13:34:58.358847 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.358831    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a0f684d-5cab-42d6-84b1-20cc2470808b" containerName="istio-proxy"
Sep 18 13:34:58.358847 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.358851    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a0f684d-5cab-42d6-84b1-20cc2470808b" containerName="istio-proxy"
Sep 18 13:34:58.361977 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.361958    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl"
Sep 18 13:34:58.412582 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.412548    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl"]
Sep 18 13:34:58.427141 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.427113    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96bc2e44-afe8-42f5-a5ab-0c8c95752ed3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl\" (UID: \"96bc2e44-afe8-42f5-a5ab-0c8c95752ed3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl"
Sep 18 13:34:58.427285 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.427246    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8d2w\" (UniqueName: \"kubernetes.io/projected/96bc2e44-afe8-42f5-a5ab-0c8c95752ed3-kube-api-access-t8d2w\") pod \"payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl\" (UID: \"96bc2e44-afe8-42f5-a5ab-0c8c95752ed3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl"
Sep 18 13:34:58.483816 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.483782    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:34:58.528172 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.528092    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8d2w\" (UniqueName: \"kubernetes.io/projected/96bc2e44-afe8-42f5-a5ab-0c8c95752ed3-kube-api-access-t8d2w\") pod \"payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl\" (UID: \"96bc2e44-afe8-42f5-a5ab-0c8c95752ed3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl"
Sep 18 13:34:58.528172 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.528153    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46mfk\" (UniqueName: \"kubernetes.io/projected/b38d31ed-2a8c-409c-9fc8-adcf2ba72730-kube-api-access-46mfk\") pod \"payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl\" (UID: \"b38d31ed-2a8c-409c-9fc8-adcf2ba72730\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl"
Sep 18 13:34:58.528386 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.528192    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96bc2e44-afe8-42f5-a5ab-0c8c95752ed3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl\" (UID: \"96bc2e44-afe8-42f5-a5ab-0c8c95752ed3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl"
Sep 18 13:34:58.528386 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.528225    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38d31ed-2a8c-409c-9fc8-adcf2ba72730-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl\" (UID: \"b38d31ed-2a8c-409c-9fc8-adcf2ba72730\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl"
Sep 18 13:34:58.528887 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.528867    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96bc2e44-afe8-42f5-a5ab-0c8c95752ed3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl\" (UID: \"96bc2e44-afe8-42f5-a5ab-0c8c95752ed3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl"
Sep 18 13:34:58.566403 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.566368    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8d2w\" (UniqueName: \"kubernetes.io/projected/96bc2e44-afe8-42f5-a5ab-0c8c95752ed3-kube-api-access-t8d2w\") pod \"payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl\" (UID: \"96bc2e44-afe8-42f5-a5ab-0c8c95752ed3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl"
Sep 18 13:34:58.580040 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.580012    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl"
Sep 18 13:34:58.629615 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.629343    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38d31ed-2a8c-409c-9fc8-adcf2ba72730-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl\" (UID: \"b38d31ed-2a8c-409c-9fc8-adcf2ba72730\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl"
Sep 18 13:34:58.629615 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.629540    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46mfk\" (UniqueName: \"kubernetes.io/projected/b38d31ed-2a8c-409c-9fc8-adcf2ba72730-kube-api-access-46mfk\") pod \"payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl\" (UID: \"b38d31ed-2a8c-409c-9fc8-adcf2ba72730\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl"
Sep 18 13:34:58.630725 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.630699    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38d31ed-2a8c-409c-9fc8-adcf2ba72730-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl\" (UID: \"b38d31ed-2a8c-409c-9fc8-adcf2ba72730\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl"
Sep 18 13:34:58.646722 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.646690    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46mfk\" (UniqueName: \"kubernetes.io/projected/b38d31ed-2a8c-409c-9fc8-adcf2ba72730-kube-api-access-46mfk\") pod \"payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl\" (UID: \"b38d31ed-2a8c-409c-9fc8-adcf2ba72730\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl"
Sep 18 13:34:58.672361 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.672314    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl"
Sep 18 13:34:58.879426 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.879395    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl"]
Sep 18 13:34:58.879583 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:34:58.879557    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96bc2e44_afe8_42f5_a5ab_0c8c95752ed3.slice/crio-731ded0e8a360a95cdadcf4ca3e4321239bf5319d039d7fd202aef9e8eb0463d WatchSource:0}: Error finding container 731ded0e8a360a95cdadcf4ca3e4321239bf5319d039d7fd202aef9e8eb0463d: Status 404 returned error can't find the container with id 731ded0e8a360a95cdadcf4ca3e4321239bf5319d039d7fd202aef9e8eb0463d
Sep 18 13:34:58.960373 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:58.960348    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl"]
Sep 18 13:34:58.962422 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:34:58.962387    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38d31ed_2a8c_409c_9fc8_adcf2ba72730.slice/crio-94494c9ea4943ec27ff74a286f488b550ed2c5942fa0c7a3ac33275e50a5cc59 WatchSource:0}: Error finding container 94494c9ea4943ec27ff74a286f488b550ed2c5942fa0c7a3ac33275e50a5cc59: Status 404 returned error can't find the container with id 94494c9ea4943ec27ff74a286f488b550ed2c5942fa0c7a3ac33275e50a5cc59
Sep 18 13:34:59.326435 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:59.326398    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl" event={"ID":"b38d31ed-2a8c-409c-9fc8-adcf2ba72730","Type":"ContainerStarted","Data":"1cdcbcf70302ead0828401706bebece7537c27565c6106bd6b5609130dadf85b"}
Sep 18 13:34:59.326928 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:59.326456    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl" event={"ID":"b38d31ed-2a8c-409c-9fc8-adcf2ba72730","Type":"ContainerStarted","Data":"94494c9ea4943ec27ff74a286f488b550ed2c5942fa0c7a3ac33275e50a5cc59"}
Sep 18 13:34:59.326928 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:59.326609    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl"
Sep 18 13:34:59.328245 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:59.328208    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl" event={"ID":"96bc2e44-afe8-42f5-a5ab-0c8c95752ed3","Type":"ContainerStarted","Data":"064cbf4250713f3e3f94d1ed1963f281f45c64c4daec6f9c2416fec68c5a5d76"}
Sep 18 13:34:59.328360 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:59.328272    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl"
Sep 18 13:34:59.328360 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:59.328298    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl" event={"ID":"96bc2e44-afe8-42f5-a5ab-0c8c95752ed3","Type":"ContainerStarted","Data":"731ded0e8a360a95cdadcf4ca3e4321239bf5319d039d7fd202aef9e8eb0463d"}
Sep 18 13:34:59.366934 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:59.366870    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl" podStartSLOduration=1.366856826 podStartE2EDuration="1.366856826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:34:59.365837666 +0000 UTC m=+1196.721297492" watchObservedRunningTime="2026-09-18 13:34:59.366856826 +0000 UTC m=+1196.722316647"
Sep 18 13:34:59.395428 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:34:59.395386    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl" podStartSLOduration=1.395374597 podStartE2EDuration="1.395374597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:34:59.395129432 +0000 UTC m=+1196.750589260" watchObservedRunningTime="2026-09-18 13:34:59.395374597 +0000 UTC m=+1196.750834418"
Sep 18 13:35:04.871969 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:04.871942    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lcqcw_1cc32441-1189-4113-bf17-2769fa23a747/ovn-acl-logging/0.log"
Sep 18 13:35:04.872366 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:04.872349    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lcqcw_1cc32441-1189-4113-bf17-2769fa23a747/ovn-acl-logging/0.log"
Sep 18 13:35:10.336107 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:10.336076    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl"
Sep 18 13:35:10.336558 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:10.336199    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl"
Sep 18 13:35:29.073052 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:29.073017    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9"]
Sep 18 13:35:29.076806 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:29.076784    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9"
Sep 18 13:35:29.080005 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:29.079986    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-92b54f-dockercfg-97v7k\""
Sep 18 13:35:29.080260 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:29.080244    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-92b54f\""
Sep 18 13:35:29.095291 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:29.095268    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9"]
Sep 18 13:35:29.120723 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:29.120694    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv"]
Sep 18 13:35:29.124261 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:29.124241    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv"
Sep 18 13:35:29.139305 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:29.139283    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv"]
Sep 18 13:35:29.222386 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:29.222347    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e08c40-2d69-494e-a0e3-aad00fb5fa6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9\" (UID: \"64e08c40-2d69-494e-a0e3-aad00fb5fa6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9"
Sep 18 13:35:29.222589 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:29.222412    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbc42\" (UniqueName: \"kubernetes.io/projected/2b873220-afcc-4185-ae47-d8b853b07971-kube-api-access-dbc42\") pod \"payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv\" (UID: \"2b873220-afcc-4185-ae47-d8b853b07971\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv"
Sep 18 13:35:29.222589 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:29.222548    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvbzz\" (UniqueName: \"kubernetes.io/projected/64e08c40-2d69-494e-a0e3-aad00fb5fa6a-kube-api-access-gvbzz\") pod \"payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9\" (UID: \"64e08c40-2d69-494e-a0e3-aad00fb5fa6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9"
Sep 18 13:35:29.222589 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:29.222581    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b873220-afcc-4185-ae47-d8b853b07971-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv\" (UID: \"2b873220-afcc-4185-ae47-d8b853b07971\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv"
Sep 18 13:35:29.323164 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:29.323065    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvbzz\" (UniqueName: \"kubernetes.io/projected/64e08c40-2d69-494e-a0e3-aad00fb5fa6a-kube-api-access-gvbzz\") pod \"payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9\" (UID: \"64e08c40-2d69-494e-a0e3-aad00fb5fa6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9"
Sep 18 13:35:29.323164 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:29.323114    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b873220-afcc-4185-ae47-d8b853b07971-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv\" (UID: \"2b873220-afcc-4185-ae47-d8b853b07971\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv"
Sep 18 13:35:29.323408 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:29.323166    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e08c40-2d69-494e-a0e3-aad00fb5fa6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9\" (UID: \"64e08c40-2d69-494e-a0e3-aad00fb5fa6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9"
Sep 18 13:35:29.323408 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:29.323306    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbc42\" (UniqueName: \"kubernetes.io/projected/2b873220-afcc-4185-ae47-d8b853b07971-kube-api-access-dbc42\") pod \"payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv\" (UID: \"2b873220-afcc-4185-ae47-d8b853b07971\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv"
Sep 18 13:35:29.323960 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:29.323938    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e08c40-2d69-494e-a0e3-aad00fb5fa6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9\" (UID: \"64e08c40-2d69-494e-a0e3-aad00fb5fa6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9"
Sep 18 13:35:29.324073 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:29.323998    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b873220-afcc-4185-ae47-d8b853b07971-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv\" (UID: \"2b873220-afcc-4185-ae47-d8b853b07971\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv"
Sep 18 13:35:29.332696 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:29.332670    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvbzz\" (UniqueName: \"kubernetes.io/projected/64e08c40-2d69-494e-a0e3-aad00fb5fa6a-kube-api-access-gvbzz\") pod \"payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9\" (UID: \"64e08c40-2d69-494e-a0e3-aad00fb5fa6a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9"
Sep 18 13:35:29.332813 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:29.332672    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbc42\" (UniqueName: \"kubernetes.io/projected/2b873220-afcc-4185-ae47-d8b853b07971-kube-api-access-dbc42\") pod \"payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv\" (UID: \"2b873220-afcc-4185-ae47-d8b853b07971\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv"
Sep 18 13:35:29.387266 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:29.387228    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9"
Sep 18 13:35:29.435095 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:29.435052    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv"
Sep 18 13:35:29.667506 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:29.667480    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9"]
Sep 18 13:35:29.670053 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:35:29.670028    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e08c40_2d69_494e_a0e3_aad00fb5fa6a.slice/crio-4a90a4fb7f5ccb98db9833e103b314d4327e73ad737a5905ed8e2021f8eb9c16 WatchSource:0}: Error finding container 4a90a4fb7f5ccb98db9833e103b314d4327e73ad737a5905ed8e2021f8eb9c16: Status 404 returned error can't find the container with id 4a90a4fb7f5ccb98db9833e103b314d4327e73ad737a5905ed8e2021f8eb9c16
Sep 18 13:35:29.734597 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:29.734571    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv"]
Sep 18 13:35:29.737626 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:35:29.737600    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b873220_afcc_4185_ae47_d8b853b07971.slice/crio-494c76dcd7b5ec9e285e11b10230523624327fa6413878fbd2060f74a48af44f WatchSource:0}: Error finding container 494c76dcd7b5ec9e285e11b10230523624327fa6413878fbd2060f74a48af44f: Status 404 returned error can't find the container with id 494c76dcd7b5ec9e285e11b10230523624327fa6413878fbd2060f74a48af44f
Sep 18 13:35:29.889964 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:29.889925    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:35:30.496638 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:30.496601    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9" event={"ID":"64e08c40-2d69-494e-a0e3-aad00fb5fa6a","Type":"ContainerStarted","Data":"6f106cc68ea5aca849fd98e90f5c0deb2382cfce8abb0e4a42d572eecec7771d"}
Sep 18 13:35:30.497136 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:30.496664    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9" event={"ID":"64e08c40-2d69-494e-a0e3-aad00fb5fa6a","Type":"ContainerStarted","Data":"4a90a4fb7f5ccb98db9833e103b314d4327e73ad737a5905ed8e2021f8eb9c16"}
Sep 18 13:35:30.497136 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:30.496718    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9"
Sep 18 13:35:30.498323 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:30.498290    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv" event={"ID":"2b873220-afcc-4185-ae47-d8b853b07971","Type":"ContainerStarted","Data":"5d2510e9511101f3c31e644d11e47ad8ac0210ca3b45db430d3d5ebaeb8b0ab1"}
Sep 18 13:35:30.498453 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:30.498338    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv" event={"ID":"2b873220-afcc-4185-ae47-d8b853b07971","Type":"ContainerStarted","Data":"494c76dcd7b5ec9e285e11b10230523624327fa6413878fbd2060f74a48af44f"}
Sep 18 13:35:30.498453 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:30.498358    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv"
Sep 18 13:35:30.517178 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:30.517131    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9" podStartSLOduration=1.517113969 podStartE2EDuration="1.517113969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:35:30.515995089 +0000 UTC m=+1227.871454915" watchObservedRunningTime="2026-09-18 13:35:30.517113969 +0000 UTC m=+1227.872573795"
Sep 18 13:35:30.537598 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:30.537550    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv" podStartSLOduration=1.537533815 podStartE2EDuration="1.537533815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:35:30.535927951 +0000 UTC m=+1227.891387770" watchObservedRunningTime="2026-09-18 13:35:30.537533815 +0000 UTC m=+1227.892993641"
Sep 18 13:35:41.507353 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:41.507317    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv"
Sep 18 13:35:41.507831 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:41.507402    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9"
Sep 18 13:35:59.147179 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:35:59.147088    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:36:04.926183 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:04.926150    2586 scope.go:117] "RemoveContainer" containerID="97693b1842c3068d669753a562492323f5ed855bf0dd17a5688e154aa263a982"
Sep 18 13:36:15.491429 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:15.491394    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj"]
Sep 18 13:36:15.496619 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:15.496599    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj"
Sep 18 13:36:15.499093 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:15.499067    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fa9905fd\""
Sep 18 13:36:15.499652 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:15.499635    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-fa9905fd-dockercfg-tkkqb\""
Sep 18 13:36:15.513481 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:15.513456    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj"]
Sep 18 13:36:15.580002 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:15.579969    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhwz7\" (UniqueName: \"kubernetes.io/projected/386d6a34-3298-4906-a11e-ab7c6bd43aef-kube-api-access-vhwz7\") pod \"payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj\" (UID: \"386d6a34-3298-4906-a11e-ab7c6bd43aef\") " pod="openshift-ingress/payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj"
Sep 18 13:36:15.580188 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:15.580028    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386d6a34-3298-4906-a11e-ab7c6bd43aef-plugins-config-volume\") pod \"payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj\" (UID: \"386d6a34-3298-4906-a11e-ab7c6bd43aef\") " pod="openshift-ingress/payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj"
Sep 18 13:36:15.681089 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:15.681059    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhwz7\" (UniqueName: \"kubernetes.io/projected/386d6a34-3298-4906-a11e-ab7c6bd43aef-kube-api-access-vhwz7\") pod \"payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj\" (UID: \"386d6a34-3298-4906-a11e-ab7c6bd43aef\") " pod="openshift-ingress/payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj"
Sep 18 13:36:15.681266 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:15.681105    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386d6a34-3298-4906-a11e-ab7c6bd43aef-plugins-config-volume\") pod \"payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj\" (UID: \"386d6a34-3298-4906-a11e-ab7c6bd43aef\") " pod="openshift-ingress/payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj"
Sep 18 13:36:15.681811 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:15.681792    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386d6a34-3298-4906-a11e-ab7c6bd43aef-plugins-config-volume\") pod \"payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj\" (UID: \"386d6a34-3298-4906-a11e-ab7c6bd43aef\") " pod="openshift-ingress/payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj"
Sep 18 13:36:15.691051 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:15.691025    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhwz7\" (UniqueName: \"kubernetes.io/projected/386d6a34-3298-4906-a11e-ab7c6bd43aef-kube-api-access-vhwz7\") pod \"payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj\" (UID: \"386d6a34-3298-4906-a11e-ab7c6bd43aef\") " pod="openshift-ingress/payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj"
Sep 18 13:36:15.807598 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:15.807500    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj"
Sep 18 13:36:16.092446 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:16.092411    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj"]
Sep 18 13:36:16.094802 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:36:16.094775    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386d6a34_3298_4906_a11e_ab7c6bd43aef.slice/crio-a950e9adbc6c0939cd943826bb8429947ec98e80a4dcd0dc4b9b0aa81fca6bc6 WatchSource:0}: Error finding container a950e9adbc6c0939cd943826bb8429947ec98e80a4dcd0dc4b9b0aa81fca6bc6: Status 404 returned error can't find the container with id a950e9adbc6c0939cd943826bb8429947ec98e80a4dcd0dc4b9b0aa81fca6bc6
Sep 18 13:36:16.752968 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:16.752932    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj" event={"ID":"386d6a34-3298-4906-a11e-ab7c6bd43aef","Type":"ContainerStarted","Data":"ae81539d36a68c65342755de017eaec3ff2967aabee459fdf1ec7011926dc5b9"}
Sep 18 13:36:16.753402 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:16.752986    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj" event={"ID":"386d6a34-3298-4906-a11e-ab7c6bd43aef","Type":"ContainerStarted","Data":"a950e9adbc6c0939cd943826bb8429947ec98e80a4dcd0dc4b9b0aa81fca6bc6"}
Sep 18 13:36:16.753402 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:16.753127    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj"
Sep 18 13:36:16.775561 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:16.775515    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj" podStartSLOduration=1.775499867 podStartE2EDuration="1.775499867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:36:16.774070383 +0000 UTC m=+1274.129530209" watchObservedRunningTime="2026-09-18 13:36:16.775499867 +0000 UTC m=+1274.130959691"
Sep 18 13:36:18.218608 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:18.218577    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2"]
Sep 18 13:36:18.218996 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:18.218765    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2" podUID="27af73d7-5496-42a7-8519-ce4724b2fb9b" containerName="payload-processing" containerID="cri-o://7f37821c53a8601ff1a13b69743313732fa38221083d272ca8d4ebc82af2cc9a" gracePeriod=30
Sep 18 13:36:18.474547 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:18.474482    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2"
Sep 18 13:36:18.510153 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:18.510117    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27af73d7-5496-42a7-8519-ce4724b2fb9b-plugins-config-volume\") pod \"27af73d7-5496-42a7-8519-ce4724b2fb9b\" (UID: \"27af73d7-5496-42a7-8519-ce4724b2fb9b\") "
Sep 18 13:36:18.510303 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:18.510189    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wz4r9\" (UniqueName: \"kubernetes.io/projected/27af73d7-5496-42a7-8519-ce4724b2fb9b-kube-api-access-wz4r9\") pod \"27af73d7-5496-42a7-8519-ce4724b2fb9b\" (UID: \"27af73d7-5496-42a7-8519-ce4724b2fb9b\") "
Sep 18 13:36:18.510572 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:18.510544    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27af73d7-5496-42a7-8519-ce4724b2fb9b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "27af73d7-5496-42a7-8519-ce4724b2fb9b" (UID: "27af73d7-5496-42a7-8519-ce4724b2fb9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:36:18.512376 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:18.512356    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27af73d7-5496-42a7-8519-ce4724b2fb9b-kube-api-access-wz4r9" (OuterVolumeSpecName: "kube-api-access-wz4r9") pod "27af73d7-5496-42a7-8519-ce4724b2fb9b" (UID: "27af73d7-5496-42a7-8519-ce4724b2fb9b"). InnerVolumeSpecName "kube-api-access-wz4r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:36:18.611123 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:18.611082    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27af73d7-5496-42a7-8519-ce4724b2fb9b-plugins-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:36:18.611123 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:18.611113    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz4r9\" (UniqueName: \"kubernetes.io/projected/27af73d7-5496-42a7-8519-ce4724b2fb9b-kube-api-access-wz4r9\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:36:18.765128 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:18.765036    2586 generic.go:356] "Generic (PLEG): container finished" podID="27af73d7-5496-42a7-8519-ce4724b2fb9b" containerID="7f37821c53a8601ff1a13b69743313732fa38221083d272ca8d4ebc82af2cc9a" exitCode=0
Sep 18 13:36:18.765128 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:18.765100    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2"
Sep 18 13:36:18.765319 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:18.765124    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2" event={"ID":"27af73d7-5496-42a7-8519-ce4724b2fb9b","Type":"ContainerDied","Data":"7f37821c53a8601ff1a13b69743313732fa38221083d272ca8d4ebc82af2cc9a"}
Sep 18 13:36:18.765319 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:18.765206    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2" event={"ID":"27af73d7-5496-42a7-8519-ce4724b2fb9b","Type":"ContainerDied","Data":"608b92e32820c21e8cf66fa7e5eaeab91b04f0f486a4349f5296d35cc260ed79"}
Sep 18 13:36:18.765319 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:18.765230    2586 scope.go:117] "RemoveContainer" containerID="7f37821c53a8601ff1a13b69743313732fa38221083d272ca8d4ebc82af2cc9a"
Sep 18 13:36:18.774695 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:18.774677    2586 scope.go:117] "RemoveContainer" containerID="7f37821c53a8601ff1a13b69743313732fa38221083d272ca8d4ebc82af2cc9a"
Sep 18 13:36:18.775055 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:36:18.775033    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f37821c53a8601ff1a13b69743313732fa38221083d272ca8d4ebc82af2cc9a\": container with ID starting with 7f37821c53a8601ff1a13b69743313732fa38221083d272ca8d4ebc82af2cc9a not found: ID does not exist" containerID="7f37821c53a8601ff1a13b69743313732fa38221083d272ca8d4ebc82af2cc9a"
Sep 18 13:36:18.775110 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:18.775064    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f37821c53a8601ff1a13b69743313732fa38221083d272ca8d4ebc82af2cc9a"} err="failed to get container status \"7f37821c53a8601ff1a13b69743313732fa38221083d272ca8d4ebc82af2cc9a\": rpc error: code = NotFound desc = could not find container \"7f37821c53a8601ff1a13b69743313732fa38221083d272ca8d4ebc82af2cc9a\": container with ID starting with 7f37821c53a8601ff1a13b69743313732fa38221083d272ca8d4ebc82af2cc9a not found: ID does not exist"
Sep 18 13:36:18.796471 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:18.796444    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2"]
Sep 18 13:36:18.810972 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:18.810947    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-34aeca-7dd8c4f7b7-8gqb2"]
Sep 18 13:36:18.850057 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:18.850018    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27af73d7-5496-42a7-8519-ce4724b2fb9b" path="/var/lib/kubelet/pods/27af73d7-5496-42a7-8519-ce4724b2fb9b/volumes"
Sep 18 13:36:27.761973 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:27.761935    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj"
Sep 18 13:36:29.911477 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:29.911439    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-021eb8-s4mxt"]
Sep 18 13:36:29.912090 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:29.912066    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27af73d7-5496-42a7-8519-ce4724b2fb9b" containerName="payload-processing"
Sep 18 13:36:29.912090 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:29.912091    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="27af73d7-5496-42a7-8519-ce4724b2fb9b" containerName="payload-processing"
Sep 18 13:36:29.912234 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:29.912217    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="27af73d7-5496-42a7-8519-ce4724b2fb9b" containerName="payload-processing"
Sep 18 13:36:29.915622 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:29.915597    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-021eb8-s4mxt"
Sep 18 13:36:29.923331 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:29.923308    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nh2cd\""
Sep 18 13:36:29.923331 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:29.923334    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:36:29.926434 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:29.926411    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-021eb8-s4mxt"]
Sep 18 13:36:30.015914 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:30.015870    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8db6d63-83cf-4f18-ac36-08b1dba6ecf8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-021eb8-s4mxt\" (UID: \"f8db6d63-83cf-4f18-ac36-08b1dba6ecf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-021eb8-s4mxt"
Sep 18 13:36:30.117028 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:30.117002    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8db6d63-83cf-4f18-ac36-08b1dba6ecf8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-021eb8-s4mxt\" (UID: \"f8db6d63-83cf-4f18-ac36-08b1dba6ecf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-021eb8-s4mxt"
Sep 18 13:36:30.117595 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:30.117576    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8db6d63-83cf-4f18-ac36-08b1dba6ecf8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-021eb8-s4mxt\" (UID: \"f8db6d63-83cf-4f18-ac36-08b1dba6ecf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-021eb8-s4mxt"
Sep 18 13:36:30.227444 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:30.227372    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-021eb8-s4mxt"
Sep 18 13:36:30.513359 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:30.513262    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-021eb8-s4mxt"]
Sep 18 13:36:30.516085 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:36:30.516046    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8db6d63_83cf_4f18_ac36_08b1dba6ecf8.slice/crio-d6a45137b4fb77f6c968f6d0cf189cecbdb93a21ec25d97a3abb82354aa0377b WatchSource:0}: Error finding container d6a45137b4fb77f6c968f6d0cf189cecbdb93a21ec25d97a3abb82354aa0377b: Status 404 returned error can't find the container with id d6a45137b4fb77f6c968f6d0cf189cecbdb93a21ec25d97a3abb82354aa0377b
Sep 18 13:36:30.830469 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:30.830440    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-021eb8-s4mxt" event={"ID":"f8db6d63-83cf-4f18-ac36-08b1dba6ecf8","Type":"ContainerStarted","Data":"94a4f49c5b006d8146121d4e3b0fc95505e508eba3c45473f1f60881c7ad4a3b"}
Sep 18 13:36:30.830579 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:30.830489    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-021eb8-s4mxt" event={"ID":"f8db6d63-83cf-4f18-ac36-08b1dba6ecf8","Type":"ContainerStarted","Data":"d6a45137b4fb77f6c968f6d0cf189cecbdb93a21ec25d97a3abb82354aa0377b"}
Sep 18 13:36:30.859604 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:30.859559    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-021eb8-s4mxt" podStartSLOduration=1.8595451060000001 podStartE2EDuration="1.859545106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:36:30.855867371 +0000 UTC m=+1288.211327209" watchObservedRunningTime="2026-09-18 13:36:30.859545106 +0000 UTC m=+1288.215004931"
Sep 18 13:36:31.837751 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:31.837715    2586 generic.go:356] "Generic (PLEG): container finished" podID="f8db6d63-83cf-4f18-ac36-08b1dba6ecf8" containerID="94a4f49c5b006d8146121d4e3b0fc95505e508eba3c45473f1f60881c7ad4a3b" exitCode=0
Sep 18 13:36:31.838157 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:31.837767    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-021eb8-s4mxt" event={"ID":"f8db6d63-83cf-4f18-ac36-08b1dba6ecf8","Type":"ContainerDied","Data":"94a4f49c5b006d8146121d4e3b0fc95505e508eba3c45473f1f60881c7ad4a3b"}
Sep 18 13:36:32.600323 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:32.600292    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:36:32.982108 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:32.982084    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-021eb8-s4mxt"
Sep 18 13:36:33.047768 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:33.047737    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8db6d63-83cf-4f18-ac36-08b1dba6ecf8-maas-api-service-ca\") pod \"f8db6d63-83cf-4f18-ac36-08b1dba6ecf8\" (UID: \"f8db6d63-83cf-4f18-ac36-08b1dba6ecf8\") "
Sep 18 13:36:33.048183 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:33.048087    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8db6d63-83cf-4f18-ac36-08b1dba6ecf8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8db6d63-83cf-4f18-ac36-08b1dba6ecf8" (UID: "f8db6d63-83cf-4f18-ac36-08b1dba6ecf8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:36:33.048257 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:33.048202    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8db6d63-83cf-4f18-ac36-08b1dba6ecf8-maas-api-service-ca\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:36:33.849991 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:33.849959    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-021eb8-s4mxt"
Sep 18 13:36:33.850171 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:33.849994    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-021eb8-s4mxt" event={"ID":"f8db6d63-83cf-4f18-ac36-08b1dba6ecf8","Type":"ContainerDied","Data":"d6a45137b4fb77f6c968f6d0cf189cecbdb93a21ec25d97a3abb82354aa0377b"}
Sep 18 13:36:33.850171 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:33.850043    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6a45137b4fb77f6c968f6d0cf189cecbdb93a21ec25d97a3abb82354aa0377b"
Sep 18 13:36:35.110673 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:35.110633    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"]
Sep 18 13:36:35.111159 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:35.110976    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg" podUID="1a939872-c16d-4d63-b20e-7f2ae7dc28cf" containerName="istio-proxy" containerID="cri-o://0c626262110dcd2a249ddfb03742cd8f5e70ddfffe041b52924f93003f142f20" gracePeriod=30
Sep 18 13:36:36.276022 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:36.275978    2586 patch_prober.go:28] interesting pod/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 13:36:36.276531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:36.276044    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg" podUID="1a939872-c16d-4d63-b20e-7f2ae7dc28cf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 13:36:40.371576 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.371502    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:36:40.525266 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.525233    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-workload-certs\") pod \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") "
Sep 18 13:36:40.525266 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.525270    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istio-podinfo\") pod \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") "
Sep 18 13:36:40.525518 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.525314    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wq24r\" (UniqueName: \"kubernetes.io/projected/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-kube-api-access-wq24r\") pod \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") "
Sep 18 13:36:40.525518 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.525340    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-workload-socket\") pod \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") "
Sep 18 13:36:40.525518 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.525375    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istio-data\") pod \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") "
Sep 18 13:36:40.525518 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.525398    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-credential-socket\") pod \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") "
Sep 18 13:36:40.525518 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.525491    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istio-token\") pod \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") "
Sep 18 13:36:40.525786 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.525524    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istio-envoy\") pod \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") "
Sep 18 13:36:40.525786 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.525526    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1a939872-c16d-4d63-b20e-7f2ae7dc28cf" (UID: "1a939872-c16d-4d63-b20e-7f2ae7dc28cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:40.525786 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.525579    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istiod-ca-cert\") pod \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\" (UID: \"1a939872-c16d-4d63-b20e-7f2ae7dc28cf\") "
Sep 18 13:36:40.525786 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.525699    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1a939872-c16d-4d63-b20e-7f2ae7dc28cf" (UID: "1a939872-c16d-4d63-b20e-7f2ae7dc28cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:40.525786 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.525717    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istio-data" (OuterVolumeSpecName: "istio-data") pod "1a939872-c16d-4d63-b20e-7f2ae7dc28cf" (UID: "1a939872-c16d-4d63-b20e-7f2ae7dc28cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:40.525786 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.525750    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1a939872-c16d-4d63-b20e-7f2ae7dc28cf" (UID: "1a939872-c16d-4d63-b20e-7f2ae7dc28cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:40.526137 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.526006    2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-workload-certs\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:36:40.526137 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.526027    2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-workload-socket\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:36:40.526137 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.526041    2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istio-data\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:36:40.526137 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.526048    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1a939872-c16d-4d63-b20e-7f2ae7dc28cf" (UID: "1a939872-c16d-4d63-b20e-7f2ae7dc28cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:36:40.526137 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.526054    2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-credential-socket\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:36:40.527735 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.527716    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1a939872-c16d-4d63-b20e-7f2ae7dc28cf" (UID: "1a939872-c16d-4d63-b20e-7f2ae7dc28cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:36:40.527809 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.527795    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1a939872-c16d-4d63-b20e-7f2ae7dc28cf" (UID: "1a939872-c16d-4d63-b20e-7f2ae7dc28cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 13:36:40.528346 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.528322    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istio-token" (OuterVolumeSpecName: "istio-token") pod "1a939872-c16d-4d63-b20e-7f2ae7dc28cf" (UID: "1a939872-c16d-4d63-b20e-7f2ae7dc28cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:36:40.528413 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.528329    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-kube-api-access-wq24r" (OuterVolumeSpecName: "kube-api-access-wq24r") pod "1a939872-c16d-4d63-b20e-7f2ae7dc28cf" (UID: "1a939872-c16d-4d63-b20e-7f2ae7dc28cf"). InnerVolumeSpecName "kube-api-access-wq24r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:36:40.627129 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.627057    2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istio-podinfo\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:36:40.627129 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.627084    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq24r\" (UniqueName: \"kubernetes.io/projected/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-kube-api-access-wq24r\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:36:40.627129 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.627096    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istio-token\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:36:40.627129 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.627104    2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istio-envoy\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:36:40.627129 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.627113    2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a939872-c16d-4d63-b20e-7f2ae7dc28cf-istiod-ca-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:36:40.865308 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.865280    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-021eb8-29828970-h4css"]
Sep 18 13:36:40.875147 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.875122    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-021eb8-29828970-h4css"]
Sep 18 13:36:40.888358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.888327    2586 generic.go:356] "Generic (PLEG): container finished" podID="1a939872-c16d-4d63-b20e-7f2ae7dc28cf" containerID="0c626262110dcd2a249ddfb03742cd8f5e70ddfffe041b52924f93003f142f20" exitCode=0
Sep 18 13:36:40.888474 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.888375    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"
Sep 18 13:36:40.888474 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.888408    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg" event={"ID":"1a939872-c16d-4d63-b20e-7f2ae7dc28cf","Type":"ContainerDied","Data":"0c626262110dcd2a249ddfb03742cd8f5e70ddfffe041b52924f93003f142f20"}
Sep 18 13:36:40.888474 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.888467    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg" event={"ID":"1a939872-c16d-4d63-b20e-7f2ae7dc28cf","Type":"ContainerDied","Data":"433ef2fb56ffaac61e10affc3cd55eca5d0b62a872ea8d91b4fed606a8e2d4f3"}
Sep 18 13:36:40.888621 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.888490    2586 scope.go:117] "RemoveContainer" containerID="0c626262110dcd2a249ddfb03742cd8f5e70ddfffe041b52924f93003f142f20"
Sep 18 13:36:40.899174 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.899139    2586 scope.go:117] "RemoveContainer" containerID="0c626262110dcd2a249ddfb03742cd8f5e70ddfffe041b52924f93003f142f20"
Sep 18 13:36:40.899553 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:36:40.899512    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c626262110dcd2a249ddfb03742cd8f5e70ddfffe041b52924f93003f142f20\": container with ID starting with 0c626262110dcd2a249ddfb03742cd8f5e70ddfffe041b52924f93003f142f20 not found: ID does not exist" containerID="0c626262110dcd2a249ddfb03742cd8f5e70ddfffe041b52924f93003f142f20"
Sep 18 13:36:40.899643 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.899553    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c626262110dcd2a249ddfb03742cd8f5e70ddfffe041b52924f93003f142f20"} err="failed to get container status \"0c626262110dcd2a249ddfb03742cd8f5e70ddfffe041b52924f93003f142f20\": rpc error: code = NotFound desc = could not find container \"0c626262110dcd2a249ddfb03742cd8f5e70ddfffe041b52924f93003f142f20\": container with ID starting with 0c626262110dcd2a249ddfb03742cd8f5e70ddfffe041b52924f93003f142f20 not found: ID does not exist"
Sep 18 13:36:40.913916 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.913868    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn"]
Sep 18 13:36:40.914176 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.914144    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn" podUID="fd3d2aa6-f631-4a99-a3e7-be8ef628d1af" containerName="payload-processing" containerID="cri-o://8f30e62871ffdd6bfbf09df65457bb99d54885392d737a74dfd98ab5a9410fc7" gracePeriod=30
Sep 18 13:36:40.920599 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.920570    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"]
Sep 18 13:36:40.925395 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.925353    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-b81e47-openshift-default-7567df9d6b-wwncg"]
Sep 18 13:36:40.928575 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.928552    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f"]
Sep 18 13:36:40.928766 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:40.928746    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f" podUID="ce7f3db6-ec3d-4817-89de-ce9aa8a62d24" containerName="payload-pre-processing" containerID="cri-o://9fec4614a9a04dcb5027795e3af0f8b5ca09acb132e44e9f1d0d08beb4b1afd4" gracePeriod=30
Sep 18 13:36:41.169460 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.169427    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-021eb8-s4mxt"]
Sep 18 13:36:41.176224 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.176194    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-021eb8-s4mxt"]
Sep 18 13:36:41.219983 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.219962    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn"
Sep 18 13:36:41.222747 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.222726    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f"
Sep 18 13:36:41.332435 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.332397    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxkm8\" (UniqueName: \"kubernetes.io/projected/fd3d2aa6-f631-4a99-a3e7-be8ef628d1af-kube-api-access-fxkm8\") pod \"fd3d2aa6-f631-4a99-a3e7-be8ef628d1af\" (UID: \"fd3d2aa6-f631-4a99-a3e7-be8ef628d1af\") "
Sep 18 13:36:41.332612 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.332470    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkjhr\" (UniqueName: \"kubernetes.io/projected/ce7f3db6-ec3d-4817-89de-ce9aa8a62d24-kube-api-access-jkjhr\") pod \"ce7f3db6-ec3d-4817-89de-ce9aa8a62d24\" (UID: \"ce7f3db6-ec3d-4817-89de-ce9aa8a62d24\") "
Sep 18 13:36:41.332612 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.332547    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd3d2aa6-f631-4a99-a3e7-be8ef628d1af-plugins-config-volume\") pod \"fd3d2aa6-f631-4a99-a3e7-be8ef628d1af\" (UID: \"fd3d2aa6-f631-4a99-a3e7-be8ef628d1af\") "
Sep 18 13:36:41.332612 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.332607    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce7f3db6-ec3d-4817-89de-ce9aa8a62d24-plugins-config-volume\") pod \"ce7f3db6-ec3d-4817-89de-ce9aa8a62d24\" (UID: \"ce7f3db6-ec3d-4817-89de-ce9aa8a62d24\") "
Sep 18 13:36:41.332942 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.332892    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd3d2aa6-f631-4a99-a3e7-be8ef628d1af-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd3d2aa6-f631-4a99-a3e7-be8ef628d1af" (UID: "fd3d2aa6-f631-4a99-a3e7-be8ef628d1af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:36:41.333076 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.332970    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce7f3db6-ec3d-4817-89de-ce9aa8a62d24-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ce7f3db6-ec3d-4817-89de-ce9aa8a62d24" (UID: "ce7f3db6-ec3d-4817-89de-ce9aa8a62d24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:36:41.334807 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.334772    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce7f3db6-ec3d-4817-89de-ce9aa8a62d24-kube-api-access-jkjhr" (OuterVolumeSpecName: "kube-api-access-jkjhr") pod "ce7f3db6-ec3d-4817-89de-ce9aa8a62d24" (UID: "ce7f3db6-ec3d-4817-89de-ce9aa8a62d24"). InnerVolumeSpecName "kube-api-access-jkjhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:36:41.334937 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.334837    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3d2aa6-f631-4a99-a3e7-be8ef628d1af-kube-api-access-fxkm8" (OuterVolumeSpecName: "kube-api-access-fxkm8") pod "fd3d2aa6-f631-4a99-a3e7-be8ef628d1af" (UID: "fd3d2aa6-f631-4a99-a3e7-be8ef628d1af"). InnerVolumeSpecName "kube-api-access-fxkm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:36:41.434406 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.434304    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd3d2aa6-f631-4a99-a3e7-be8ef628d1af-plugins-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:36:41.434406 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.434355    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce7f3db6-ec3d-4817-89de-ce9aa8a62d24-plugins-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:36:41.434406 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.434364    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxkm8\" (UniqueName: \"kubernetes.io/projected/fd3d2aa6-f631-4a99-a3e7-be8ef628d1af-kube-api-access-fxkm8\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:36:41.434406 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.434374    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkjhr\" (UniqueName: \"kubernetes.io/projected/ce7f3db6-ec3d-4817-89de-ce9aa8a62d24-kube-api-access-jkjhr\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:36:41.895153 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.895122    2586 generic.go:356] "Generic (PLEG): container finished" podID="ce7f3db6-ec3d-4817-89de-ce9aa8a62d24" containerID="9fec4614a9a04dcb5027795e3af0f8b5ca09acb132e44e9f1d0d08beb4b1afd4" exitCode=0
Sep 18 13:36:41.895355 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.895174    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f"
Sep 18 13:36:41.895355 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.895216    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f" event={"ID":"ce7f3db6-ec3d-4817-89de-ce9aa8a62d24","Type":"ContainerDied","Data":"9fec4614a9a04dcb5027795e3af0f8b5ca09acb132e44e9f1d0d08beb4b1afd4"}
Sep 18 13:36:41.895355 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.895278    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f" event={"ID":"ce7f3db6-ec3d-4817-89de-ce9aa8a62d24","Type":"ContainerDied","Data":"4a9c397b1cc572d07c2ea83063cc12d98841bf15c3766d3c81699969be7e90b7"}
Sep 18 13:36:41.895355 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.895312    2586 scope.go:117] "RemoveContainer" containerID="9fec4614a9a04dcb5027795e3af0f8b5ca09acb132e44e9f1d0d08beb4b1afd4"
Sep 18 13:36:41.897807 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.897781    2586 generic.go:356] "Generic (PLEG): container finished" podID="fd3d2aa6-f631-4a99-a3e7-be8ef628d1af" containerID="8f30e62871ffdd6bfbf09df65457bb99d54885392d737a74dfd98ab5a9410fc7" exitCode=0
Sep 18 13:36:41.897962 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.897818    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn" event={"ID":"fd3d2aa6-f631-4a99-a3e7-be8ef628d1af","Type":"ContainerDied","Data":"8f30e62871ffdd6bfbf09df65457bb99d54885392d737a74dfd98ab5a9410fc7"}
Sep 18 13:36:41.897962 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.897832    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn"
Sep 18 13:36:41.897962 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.897867    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn" event={"ID":"fd3d2aa6-f631-4a99-a3e7-be8ef628d1af","Type":"ContainerDied","Data":"cbfa5c1f8a497265b76fd6086974d87ae436600a722c74c99afb3de6adc24abf"}
Sep 18 13:36:41.905390 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.905359    2586 scope.go:117] "RemoveContainer" containerID="9fec4614a9a04dcb5027795e3af0f8b5ca09acb132e44e9f1d0d08beb4b1afd4"
Sep 18 13:36:41.905731 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:36:41.905700    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fec4614a9a04dcb5027795e3af0f8b5ca09acb132e44e9f1d0d08beb4b1afd4\": container with ID starting with 9fec4614a9a04dcb5027795e3af0f8b5ca09acb132e44e9f1d0d08beb4b1afd4 not found: ID does not exist" containerID="9fec4614a9a04dcb5027795e3af0f8b5ca09acb132e44e9f1d0d08beb4b1afd4"
Sep 18 13:36:41.905832 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.905741    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fec4614a9a04dcb5027795e3af0f8b5ca09acb132e44e9f1d0d08beb4b1afd4"} err="failed to get container status \"9fec4614a9a04dcb5027795e3af0f8b5ca09acb132e44e9f1d0d08beb4b1afd4\": rpc error: code = NotFound desc = could not find container \"9fec4614a9a04dcb5027795e3af0f8b5ca09acb132e44e9f1d0d08beb4b1afd4\": container with ID starting with 9fec4614a9a04dcb5027795e3af0f8b5ca09acb132e44e9f1d0d08beb4b1afd4 not found: ID does not exist"
Sep 18 13:36:41.905832 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.905766    2586 scope.go:117] "RemoveContainer" containerID="8f30e62871ffdd6bfbf09df65457bb99d54885392d737a74dfd98ab5a9410fc7"
Sep 18 13:36:41.915097 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.915083    2586 scope.go:117] "RemoveContainer" containerID="8f30e62871ffdd6bfbf09df65457bb99d54885392d737a74dfd98ab5a9410fc7"
Sep 18 13:36:41.915388 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:36:41.915357    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f30e62871ffdd6bfbf09df65457bb99d54885392d737a74dfd98ab5a9410fc7\": container with ID starting with 8f30e62871ffdd6bfbf09df65457bb99d54885392d737a74dfd98ab5a9410fc7 not found: ID does not exist" containerID="8f30e62871ffdd6bfbf09df65457bb99d54885392d737a74dfd98ab5a9410fc7"
Sep 18 13:36:41.915477 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.915391    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f30e62871ffdd6bfbf09df65457bb99d54885392d737a74dfd98ab5a9410fc7"} err="failed to get container status \"8f30e62871ffdd6bfbf09df65457bb99d54885392d737a74dfd98ab5a9410fc7\": rpc error: code = NotFound desc = could not find container \"8f30e62871ffdd6bfbf09df65457bb99d54885392d737a74dfd98ab5a9410fc7\": container with ID starting with 8f30e62871ffdd6bfbf09df65457bb99d54885392d737a74dfd98ab5a9410fc7 not found: ID does not exist"
Sep 18 13:36:41.920527 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.920492    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f"]
Sep 18 13:36:41.924253 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.924233    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-021eb8-788b4fc4f4-8x25f"]
Sep 18 13:36:41.940450 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.940427    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn"]
Sep 18 13:36:41.944071 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:41.944043    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-021eb8-5b985b5bf9-wz9fn"]
Sep 18 13:36:42.849920 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:42.849865    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a939872-c16d-4d63-b20e-7f2ae7dc28cf" path="/var/lib/kubelet/pods/1a939872-c16d-4d63-b20e-7f2ae7dc28cf/volumes"
Sep 18 13:36:42.850401 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:42.850325    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b120ea14-f6cb-41ce-afe4-33b41cf7a44c" path="/var/lib/kubelet/pods/b120ea14-f6cb-41ce-afe4-33b41cf7a44c/volumes"
Sep 18 13:36:42.850683 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:42.850664    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce7f3db6-ec3d-4817-89de-ce9aa8a62d24" path="/var/lib/kubelet/pods/ce7f3db6-ec3d-4817-89de-ce9aa8a62d24/volumes"
Sep 18 13:36:42.851024 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:42.851007    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8db6d63-83cf-4f18-ac36-08b1dba6ecf8" path="/var/lib/kubelet/pods/f8db6d63-83cf-4f18-ac36-08b1dba6ecf8/volumes"
Sep 18 13:36:42.851678 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:42.851659    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd3d2aa6-f631-4a99-a3e7-be8ef628d1af" path="/var/lib/kubelet/pods/fd3d2aa6-f631-4a99-a3e7-be8ef628d1af/volumes"
Sep 18 13:36:50.328782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.328739    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"]
Sep 18 13:36:50.329350 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.329331    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce7f3db6-ec3d-4817-89de-ce9aa8a62d24" containerName="payload-pre-processing"
Sep 18 13:36:50.329444 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.329351    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce7f3db6-ec3d-4817-89de-ce9aa8a62d24" containerName="payload-pre-processing"
Sep 18 13:36:50.329444 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.329381    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd3d2aa6-f631-4a99-a3e7-be8ef628d1af" containerName="payload-processing"
Sep 18 13:36:50.329444 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.329389    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd3d2aa6-f631-4a99-a3e7-be8ef628d1af" containerName="payload-processing"
Sep 18 13:36:50.329444 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.329412    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a939872-c16d-4d63-b20e-7f2ae7dc28cf" containerName="istio-proxy"
Sep 18 13:36:50.329444 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.329420    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a939872-c16d-4d63-b20e-7f2ae7dc28cf" containerName="istio-proxy"
Sep 18 13:36:50.329444 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.329429    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8db6d63-83cf-4f18-ac36-08b1dba6ecf8" containerName="revoke-keys"
Sep 18 13:36:50.329444 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.329437    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8db6d63-83cf-4f18-ac36-08b1dba6ecf8" containerName="revoke-keys"
Sep 18 13:36:50.329782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.329531    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8db6d63-83cf-4f18-ac36-08b1dba6ecf8" containerName="revoke-keys"
Sep 18 13:36:50.329782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.329543    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce7f3db6-ec3d-4817-89de-ce9aa8a62d24" containerName="payload-pre-processing"
Sep 18 13:36:50.329782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.329555    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd3d2aa6-f631-4a99-a3e7-be8ef628d1af" containerName="payload-processing"
Sep 18 13:36:50.329782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.329566    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a939872-c16d-4d63-b20e-7f2ae7dc28cf" containerName="istio-proxy"
Sep 18 13:36:50.334651 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.334624    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.337202 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.337179    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-537abe-openshift-default-dockercfg-t7ktj\""
Sep 18 13:36:50.337500 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.337484    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 13:36:50.385604 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.385573    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"]
Sep 18 13:36:50.427590 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.427550    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-credential-socket\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.427738 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.427600    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-workload-certs\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.427782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.427752    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e004867-2880-4eda-8770-f1e6fb00eb8d-istio-podinfo\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.427818 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.427785    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smkjc\" (UniqueName: \"kubernetes.io/projected/7e004867-2880-4eda-8770-f1e6fb00eb8d-kube-api-access-smkjc\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.427818 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.427805    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e004867-2880-4eda-8770-f1e6fb00eb8d-istio-token\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.427895 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.427880    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-istio-data\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.427963 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.427945    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-istio-envoy\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.428002 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.427973    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e004867-2880-4eda-8770-f1e6fb00eb8d-istiod-ca-cert\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.428056 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.428041    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-workload-socket\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.528966 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.528929    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e004867-2880-4eda-8770-f1e6fb00eb8d-istio-podinfo\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.529179 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.528979    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smkjc\" (UniqueName: \"kubernetes.io/projected/7e004867-2880-4eda-8770-f1e6fb00eb8d-kube-api-access-smkjc\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.529179 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.528997    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e004867-2880-4eda-8770-f1e6fb00eb8d-istio-token\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.529179 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.529031    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-istio-data\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.529179 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.529068    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-istio-envoy\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.529179 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.529095    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e004867-2880-4eda-8770-f1e6fb00eb8d-istiod-ca-cert\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.529179 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.529144    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-workload-socket\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.529179 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.529178    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-credential-socket\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.529583 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.529201    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-workload-certs\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.529583 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.529463    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-istio-data\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.529693 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.529588    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-workload-certs\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.529748 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.529722    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-credential-socket\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.529833 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.529789    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-workload-socket\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.529892 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.529870    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e004867-2880-4eda-8770-f1e6fb00eb8d-istiod-ca-cert\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.532378 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.532352    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-istio-envoy\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.532570 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.532555    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e004867-2880-4eda-8770-f1e6fb00eb8d-istio-podinfo\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.541727 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.541704    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e004867-2880-4eda-8770-f1e6fb00eb8d-istio-token\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.542255 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.542233    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smkjc\" (UniqueName: \"kubernetes.io/projected/7e004867-2880-4eda-8770-f1e6fb00eb8d-kube-api-access-smkjc\") pod \"e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") " pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.645648 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.645615    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:50.948053 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.948023    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"]
Sep 18 13:36:50.948200 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:36:50.948173    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e004867_2880_4eda_8770_f1e6fb00eb8d.slice/crio-976306bd77fc4214fa5c69fea48d4d1dfde6440bc26a1dedc125be7f64f4ec6f WatchSource:0}: Error finding container 976306bd77fc4214fa5c69fea48d4d1dfde6440bc26a1dedc125be7f64f4ec6f: Status 404 returned error can't find the container with id 976306bd77fc4214fa5c69fea48d4d1dfde6440bc26a1dedc125be7f64f4ec6f
Sep 18 13:36:50.950795 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.950764    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:36:50.950874 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.950824    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:36:50.950874 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:50.950852    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:36:51.954799 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:51.954757    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959" event={"ID":"7e004867-2880-4eda-8770-f1e6fb00eb8d","Type":"ContainerStarted","Data":"8174ba9c274f09f48b0d3a347e5a7989a886efe6c21c86b826d205548ef55119"}
Sep 18 13:36:51.955237 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:51.954837    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959" event={"ID":"7e004867-2880-4eda-8770-f1e6fb00eb8d","Type":"ContainerStarted","Data":"976306bd77fc4214fa5c69fea48d4d1dfde6440bc26a1dedc125be7f64f4ec6f"}
Sep 18 13:36:51.986331 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:51.986282    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959" podStartSLOduration=1.986265521 podStartE2EDuration="1.986265521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:36:51.985243573 +0000 UTC m=+1309.340703397" watchObservedRunningTime="2026-09-18 13:36:51.986265521 +0000 UTC m=+1309.341725346"
Sep 18 13:36:52.645982 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:52.645937    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:52.651315 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:52.651290    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:52.960538 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:52.960457    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:36:52.961556 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:36:52.961538    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:37:01.807804 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:01.807756    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:37:04.952081 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:04.952044    2586 scope.go:117] "RemoveContainer" containerID="015da423b77568bcd171be7756313cf2e41f739a60562aa60bfb5c5830e5d332"
Sep 18 13:37:22.121026 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:22.120989    2586 generic.go:356] "Generic (PLEG): container finished" podID="7e004867-2880-4eda-8770-f1e6fb00eb8d" containerID="8174ba9c274f09f48b0d3a347e5a7989a886efe6c21c86b826d205548ef55119" exitCode=0
Sep 18 13:37:22.121429 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:22.121058    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959" event={"ID":"7e004867-2880-4eda-8770-f1e6fb00eb8d","Type":"ContainerDied","Data":"8174ba9c274f09f48b0d3a347e5a7989a886efe6c21c86b826d205548ef55119"}
Sep 18 13:37:22.121532 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:22.121517    2586 scope.go:117] "RemoveContainer" containerID="8174ba9c274f09f48b0d3a347e5a7989a886efe6c21c86b826d205548ef55119"
Sep 18 13:37:22.122690 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:22.122659    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:37:22.122741 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:22.122731    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:37:22.122776 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:22.122762    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:37:22.961372 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:22.961336    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:37:23.133024 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:23.132984    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959" event={"ID":"7e004867-2880-4eda-8770-f1e6fb00eb8d","Type":"ContainerStarted","Data":"23c560272e5d33419f62287446d9b3c6ce64306f60871d8a57e417c18b8a40cf"}
Sep 18 13:37:23.646361 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:23.646285    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:37:23.647142 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:23.647118    2586 patch_prober.go:28] interesting pod/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 18 13:37:23.647233 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:23.647170    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959" podUID="7e004867-2880-4eda-8770-f1e6fb00eb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 18 13:37:24.646869 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:24.646821    2586 patch_prober.go:28] interesting pod/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 18 13:37:24.647312 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:24.646894    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959" podUID="7e004867-2880-4eda-8770-f1e6fb00eb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 18 13:37:25.079266 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.079183    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"]
Sep 18 13:37:25.081667 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.081647    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.085962 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.085940    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-c50e10-openshift-default-dockercfg-jcc6p\""
Sep 18 13:37:25.104345 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.104320    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"]
Sep 18 13:37:25.154218 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.154185    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-workload-certs\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.154218 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.154222    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-workload-socket\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.154424 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.154239    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istiod-ca-cert\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.154424 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.154262    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istio-data\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.154424 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.154346    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istio-envoy\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.154424 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.154382    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istio-token\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.154578 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.154462    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4mzd\" (UniqueName: \"kubernetes.io/projected/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-kube-api-access-x4mzd\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.154578 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.154492    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istio-podinfo\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.154578 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.154518    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-credential-socket\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.255316 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.255280    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4mzd\" (UniqueName: \"kubernetes.io/projected/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-kube-api-access-x4mzd\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.255316 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.255329    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istio-podinfo\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.255573 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.255356    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-credential-socket\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.255573 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.255390    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-workload-certs\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.255573 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.255416    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-workload-socket\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.255573 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.255439    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istiod-ca-cert\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.255573 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.255475    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istio-data\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.255573 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.255559    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istio-envoy\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.255890 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.255596    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istio-token\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.255890 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.255774    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-credential-socket\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.255890 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.255833    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-workload-certs\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.256075 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.256001    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istio-data\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.256557 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.256529    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-workload-socket\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.256849 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.256615    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istiod-ca-cert\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.258269 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.258240    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istio-podinfo\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.258397 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.258379    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istio-envoy\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.272087 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.272065    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4mzd\" (UniqueName: \"kubernetes.io/projected/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-kube-api-access-x4mzd\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.276859 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.276837    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istio-token\") pod \"e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") " pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.391537 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.391515    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:25.646508 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.646420    2586 patch_prober.go:28] interesting pod/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 18 13:37:25.646508 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.646485    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959" podUID="7e004867-2880-4eda-8770-f1e6fb00eb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 18 13:37:25.709282 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.705042    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"]
Sep 18 13:37:25.709282 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:37:25.708409    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f712e91_6f18_4fe4_9f0b_5550d6a5e368.slice/crio-01ef8180234e89f3da22527756733a7719a80723cc5ed259f0a57e4833e74da8 WatchSource:0}: Error finding container 01ef8180234e89f3da22527756733a7719a80723cc5ed259f0a57e4833e74da8: Status 404 returned error can't find the container with id 01ef8180234e89f3da22527756733a7719a80723cc5ed259f0a57e4833e74da8
Sep 18 13:37:25.714044 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.712777    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:37:25.714044 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.712861    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:37:25.714044 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:25.713609    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:37:26.154579 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:26.154535    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7" event={"ID":"5f712e91-6f18-4fe4-9f0b-5550d6a5e368","Type":"ContainerStarted","Data":"0f023bb75641e724c5216e597428665c64c6cf4313c023cb8ab820f64fbde425"}
Sep 18 13:37:26.154767 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:26.154642    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7" event={"ID":"5f712e91-6f18-4fe4-9f0b-5550d6a5e368","Type":"ContainerStarted","Data":"01ef8180234e89f3da22527756733a7719a80723cc5ed259f0a57e4833e74da8"}
Sep 18 13:37:26.190453 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:26.190387    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7" podStartSLOduration=1.190363427 podStartE2EDuration="1.190363427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:37:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:37:26.185603523 +0000 UTC m=+1343.541063369" watchObservedRunningTime="2026-09-18 13:37:26.190363427 +0000 UTC m=+1343.545823254"
Sep 18 13:37:26.391648 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:26.391610    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:26.397075 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:26.397044    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:26.611221 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:26.611125    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:37:26.647094 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:26.647051    2586 patch_prober.go:28] interesting pod/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 18 13:37:26.647268 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:26.647131    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959" podUID="7e004867-2880-4eda-8770-f1e6fb00eb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 18 13:37:27.159225 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:27.159191    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:27.160126 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:27.160102    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:27.646970 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:27.646942    2586 patch_prober.go:28] interesting pod/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 18 13:37:27.647125 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:27.646998    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959" podUID="7e004867-2880-4eda-8770-f1e6fb00eb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 18 13:37:28.696509 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:28.696470    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:37:28.697012 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:28.696870    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:37:28.698047 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:28.698012    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:37:31.279398 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:31.279362    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf"]
Sep 18 13:37:31.281924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:31.281887    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf"
Sep 18 13:37:31.289449 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:31.289420    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-c50e10\""
Sep 18 13:37:31.289567 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:31.289452    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-c50e10-dockercfg-rjv99\""
Sep 18 13:37:31.402066 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:31.402028    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf"]
Sep 18 13:37:31.427264 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:31.427231    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf\" (UID: \"8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf"
Sep 18 13:37:31.427438 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:31.427373    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl4z6\" (UniqueName: \"kubernetes.io/projected/8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b-kube-api-access-kl4z6\") pod \"payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf\" (UID: \"8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf"
Sep 18 13:37:31.528922 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:31.528863    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl4z6\" (UniqueName: \"kubernetes.io/projected/8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b-kube-api-access-kl4z6\") pod \"payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf\" (UID: \"8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf"
Sep 18 13:37:31.529096 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:31.528971    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf\" (UID: \"8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf"
Sep 18 13:37:31.529629 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:31.529583    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf\" (UID: \"8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf"
Sep 18 13:37:31.554251 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:31.554224    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl4z6\" (UniqueName: \"kubernetes.io/projected/8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b-kube-api-access-kl4z6\") pod \"payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf\" (UID: \"8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf"
Sep 18 13:37:31.592643 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:31.592608    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf"
Sep 18 13:37:31.888508 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:31.888480    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf"]
Sep 18 13:37:31.891366 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:37:31.891333    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad1326c_f8c5_4d3b_bdd1_0956529c4a7b.slice/crio-5453eb2d8e818186117818b04259aff44ad0e47d20ac41e7b343af9a5728c273 WatchSource:0}: Error finding container 5453eb2d8e818186117818b04259aff44ad0e47d20ac41e7b343af9a5728c273: Status 404 returned error can't find the container with id 5453eb2d8e818186117818b04259aff44ad0e47d20ac41e7b343af9a5728c273
Sep 18 13:37:32.186047 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:32.185953    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf" event={"ID":"8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b","Type":"ContainerStarted","Data":"ab7636577500e2b0927b41d46a9e044412191fce786b8e48ba0e68fd870c4176"}
Sep 18 13:37:32.186047 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:32.186012    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf" event={"ID":"8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b","Type":"ContainerStarted","Data":"5453eb2d8e818186117818b04259aff44ad0e47d20ac41e7b343af9a5728c273"}
Sep 18 13:37:32.186264 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:32.186118    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf"
Sep 18 13:37:32.210222 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:32.210158    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf" podStartSLOduration=1.2101363410000001 podStartE2EDuration="1.210136341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:37:32.209030363 +0000 UTC m=+1349.564490188" watchObservedRunningTime="2026-09-18 13:37:32.210136341 +0000 UTC m=+1349.565596167"
Sep 18 13:37:43.193484 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:43.193455    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf"
Sep 18 13:37:54.022973 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.022941    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"]
Sep 18 13:37:54.026525 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.026509    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:37:54.032876 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.032852    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fa9905fd\"/\"default-dockercfg-zhdv4\""
Sep 18 13:37:54.045085 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.045057    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fa9905fd\"/\"kube-root-ca.crt\""
Sep 18 13:37:54.045627 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.045607    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fa9905fd\"/\"openshift-service-ca.crt\""
Sep 18 13:37:54.045706 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.045641    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fa9905fd\"/\"e2e-lifecycle-model-fa9905fd-kserve-self-signed-certs\""
Sep 18 13:37:54.070867 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.070835    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"]
Sep 18 13:37:54.164024 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.163989    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/777420fd-638d-4703-934d-5b31b361c67a-model-cache\") pod \"e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd\" (UID: \"777420fd-638d-4703-934d-5b31b361c67a\") " pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:37:54.164216 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.164063    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/777420fd-638d-4703-934d-5b31b361c67a-home\") pod \"e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd\" (UID: \"777420fd-638d-4703-934d-5b31b361c67a\") " pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:37:54.164216 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.164111    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5vtd\" (UniqueName: \"kubernetes.io/projected/777420fd-638d-4703-934d-5b31b361c67a-kube-api-access-m5vtd\") pod \"e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd\" (UID: \"777420fd-638d-4703-934d-5b31b361c67a\") " pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:37:54.164216 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.164171    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/777420fd-638d-4703-934d-5b31b361c67a-tls-certs\") pod \"e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd\" (UID: \"777420fd-638d-4703-934d-5b31b361c67a\") " pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:37:54.164216 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.164203    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/777420fd-638d-4703-934d-5b31b361c67a-dshm\") pod \"e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd\" (UID: \"777420fd-638d-4703-934d-5b31b361c67a\") " pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:37:54.164371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.164279    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/777420fd-638d-4703-934d-5b31b361c67a-tmp-dir\") pod \"e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd\" (UID: \"777420fd-638d-4703-934d-5b31b361c67a\") " pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:37:54.265276 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.265237    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/777420fd-638d-4703-934d-5b31b361c67a-dshm\") pod \"e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd\" (UID: \"777420fd-638d-4703-934d-5b31b361c67a\") " pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:37:54.265442 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.265380    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/777420fd-638d-4703-934d-5b31b361c67a-tmp-dir\") pod \"e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd\" (UID: \"777420fd-638d-4703-934d-5b31b361c67a\") " pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:37:54.265442 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.265432    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/777420fd-638d-4703-934d-5b31b361c67a-model-cache\") pod \"e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd\" (UID: \"777420fd-638d-4703-934d-5b31b361c67a\") " pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:37:54.265536 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.265496    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/777420fd-638d-4703-934d-5b31b361c67a-home\") pod \"e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd\" (UID: \"777420fd-638d-4703-934d-5b31b361c67a\") " pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:37:54.265536 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.265523    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5vtd\" (UniqueName: \"kubernetes.io/projected/777420fd-638d-4703-934d-5b31b361c67a-kube-api-access-m5vtd\") pod \"e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd\" (UID: \"777420fd-638d-4703-934d-5b31b361c67a\") " pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:37:54.265853 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.265721    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/777420fd-638d-4703-934d-5b31b361c67a-tls-certs\") pod \"e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd\" (UID: \"777420fd-638d-4703-934d-5b31b361c67a\") " pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:37:54.265853 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.265751    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/777420fd-638d-4703-934d-5b31b361c67a-tmp-dir\") pod \"e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd\" (UID: \"777420fd-638d-4703-934d-5b31b361c67a\") " pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:37:54.265853 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.265838    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/777420fd-638d-4703-934d-5b31b361c67a-model-cache\") pod \"e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd\" (UID: \"777420fd-638d-4703-934d-5b31b361c67a\") " pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:37:54.266144 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.265881    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/777420fd-638d-4703-934d-5b31b361c67a-home\") pod \"e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd\" (UID: \"777420fd-638d-4703-934d-5b31b361c67a\") " pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:37:54.267734 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.267708    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/777420fd-638d-4703-934d-5b31b361c67a-dshm\") pod \"e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd\" (UID: \"777420fd-638d-4703-934d-5b31b361c67a\") " pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:37:54.268205 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.268187    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/777420fd-638d-4703-934d-5b31b361c67a-tls-certs\") pod \"e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd\" (UID: \"777420fd-638d-4703-934d-5b31b361c67a\") " pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:37:54.275478 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.275454    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5vtd\" (UniqueName: \"kubernetes.io/projected/777420fd-638d-4703-934d-5b31b361c67a-kube-api-access-m5vtd\") pod \"e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd\" (UID: \"777420fd-638d-4703-934d-5b31b361c67a\") " pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:37:54.336239 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.336201    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:37:54.637764 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:54.637732    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"]
Sep 18 13:37:54.639149 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:37:54.639108    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod777420fd_638d_4703_934d_5b31b361c67a.slice/crio-a7c809a6305ab97d685fabb34000455ec832cc311acf5a80cc93599f3ec25130 WatchSource:0}: Error finding container a7c809a6305ab97d685fabb34000455ec832cc311acf5a80cc93599f3ec25130: Status 404 returned error can't find the container with id a7c809a6305ab97d685fabb34000455ec832cc311acf5a80cc93599f3ec25130
Sep 18 13:37:55.308326 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:55.308277    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd" event={"ID":"777420fd-638d-4703-934d-5b31b361c67a","Type":"ContainerStarted","Data":"27568dc7195263fbd5e0dbfc9728b55c63ec5ab9b3f9f8678d03b84b91cb4892"}
Sep 18 13:37:55.308326 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:55.308327    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd" event={"ID":"777420fd-638d-4703-934d-5b31b361c67a","Type":"ContainerStarted","Data":"a7c809a6305ab97d685fabb34000455ec832cc311acf5a80cc93599f3ec25130"}
Sep 18 13:37:55.346981 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:55.346616    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd" podStartSLOduration=2.346598342 podStartE2EDuration="2.346598342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:37:55.344563767 +0000 UTC m=+1372.700023591" watchObservedRunningTime="2026-09-18 13:37:55.346598342 +0000 UTC m=+1372.702058168"
Sep 18 13:37:57.161134 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:57.161095    2586 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 18 13:37:57.161554 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:57.161152    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7" podUID="5f712e91-6f18-4fe4-9f0b-5550d6a5e368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 18 13:37:57.323122 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:57.323088    2586 generic.go:356] "Generic (PLEG): container finished" podID="5f712e91-6f18-4fe4-9f0b-5550d6a5e368" containerID="0f023bb75641e724c5216e597428665c64c6cf4313c023cb8ab820f64fbde425" exitCode=0
Sep 18 13:37:57.323281 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:57.323164    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7" event={"ID":"5f712e91-6f18-4fe4-9f0b-5550d6a5e368","Type":"ContainerDied","Data":"0f023bb75641e724c5216e597428665c64c6cf4313c023cb8ab820f64fbde425"}
Sep 18 13:37:57.323593 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:57.323577    2586 scope.go:117] "RemoveContainer" containerID="0f023bb75641e724c5216e597428665c64c6cf4313c023cb8ab820f64fbde425"
Sep 18 13:37:57.324805 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:57.324779    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:37:57.324865 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:57.324848    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:37:57.324922 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:57.324879    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:37:58.333544 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:58.333510    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7" event={"ID":"5f712e91-6f18-4fe4-9f0b-5550d6a5e368","Type":"ContainerStarted","Data":"968fe0f23546141ede6d094fb4c361640a2d335b9af196766b69221871be78e3"}
Sep 18 13:37:58.392063 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:58.392032    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:37:58.392256 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:58.392236    2586 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 18 13:37:58.392301 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:58.392286    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7" podUID="5f712e91-6f18-4fe4-9f0b-5550d6a5e368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 18 13:37:59.392387 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:59.392351    2586 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 18 13:37:59.392782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:37:59.392403    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7" podUID="5f712e91-6f18-4fe4-9f0b-5550d6a5e368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 18 13:38:00.391988 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:00.391954    2586 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 18 13:38:00.392158 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:00.392007    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7" podUID="5f712e91-6f18-4fe4-9f0b-5550d6a5e368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 18 13:38:01.392173 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:01.392138    2586 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 18 13:38:01.392573 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:01.392192    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7" podUID="5f712e91-6f18-4fe4-9f0b-5550d6a5e368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 18 13:38:02.392318 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:02.392283    2586 patch_prober.go:28] interesting pod/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 18 13:38:02.392731 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:02.392338    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7" podUID="5f712e91-6f18-4fe4-9f0b-5550d6a5e368" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 18 13:38:03.395640 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:03.395612    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:38:03.396025 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:03.395955    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:38:03.396658 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:03.396636    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:38:04.337217 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:04.337186    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:38:04.337515 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:04.337489    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:38:04.350358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:04.350333    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:38:04.375252 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:04.375224    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:38:18.352458 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:18.352419    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:38:36.980476 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:36.980437    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"]
Sep 18 13:38:36.984407 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:36.984390    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:38:36.986679 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:36.986662    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-537abe\"/\"auth-test-model-537abe-kserve-self-signed-certs\""
Sep 18 13:38:36.987309 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:36.987292    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-537abe\"/\"default-dockercfg-vw4dd\""
Sep 18 13:38:36.987374 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:36.987359    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-537abe\"/\"openshift-service-ca.crt\""
Sep 18 13:38:36.988575 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:36.988555    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-537abe\"/\"kube-root-ca.crt\""
Sep 18 13:38:37.001410 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:37.001386    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"]
Sep 18 13:38:37.088412 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:37.088369    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd6bx\" (UniqueName: \"kubernetes.io/projected/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-kube-api-access-zd6bx\") pod \"auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq\" (UID: \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\") " pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:38:37.088581 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:37.088451    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-tmp-dir\") pod \"auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq\" (UID: \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\") " pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:38:37.088581 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:37.088477    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-tls-certs\") pod \"auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq\" (UID: \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\") " pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:38:37.088581 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:37.088500    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-home\") pod \"auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq\" (UID: \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\") " pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:38:37.088581 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:37.088514    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-dshm\") pod \"auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq\" (UID: \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\") " pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:38:37.088719 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:37.088625    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-model-cache\") pod \"auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq\" (UID: \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\") " pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:38:37.189382 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:37.189327    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-tmp-dir\") pod \"auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq\" (UID: \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\") " pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:38:37.189382 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:37.189388    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-tls-certs\") pod \"auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq\" (UID: \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\") " pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:38:37.189614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:37.189413    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-home\") pod \"auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq\" (UID: \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\") " pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:38:37.189614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:37.189427    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-dshm\") pod \"auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq\" (UID: \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\") " pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:38:37.189614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:37.189481    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-model-cache\") pod \"auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq\" (UID: \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\") " pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:38:37.189614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:37.189540    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd6bx\" (UniqueName: \"kubernetes.io/projected/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-kube-api-access-zd6bx\") pod \"auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq\" (UID: \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\") " pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:38:37.189794 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:37.189768    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-tmp-dir\") pod \"auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq\" (UID: \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\") " pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:38:37.189832 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:37.189803    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-home\") pod \"auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq\" (UID: \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\") " pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:38:37.189933 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:37.189878    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-model-cache\") pod \"auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq\" (UID: \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\") " pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:38:37.191839 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:37.191811    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-dshm\") pod \"auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq\" (UID: \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\") " pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:38:37.192106 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:37.192090    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-tls-certs\") pod \"auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq\" (UID: \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\") " pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:38:37.197715 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:37.197696    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd6bx\" (UniqueName: \"kubernetes.io/projected/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-kube-api-access-zd6bx\") pod \"auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq\" (UID: \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\") " pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:38:37.293188 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:37.293105    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:38:37.570283 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:37.570163    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"]
Sep 18 13:38:37.574512 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:38:37.574484    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2a929d1_6dd1_4429_b1c9_a952ee5e71b6.slice/crio-0f64334d14f4ba936be647beae06171f174a1890f747ab201bd25ccd53a37b73 WatchSource:0}: Error finding container 0f64334d14f4ba936be647beae06171f174a1890f747ab201bd25ccd53a37b73: Status 404 returned error can't find the container with id 0f64334d14f4ba936be647beae06171f174a1890f747ab201bd25ccd53a37b73
Sep 18 13:38:38.535517 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:38.535479    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq" event={"ID":"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6","Type":"ContainerStarted","Data":"2d86d4752aeb88c15e28904a056fa7f1e7b5185dc300fd2c5b41a2232f1a802f"}
Sep 18 13:38:38.536013 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:38.535525    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq" event={"ID":"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6","Type":"ContainerStarted","Data":"0f64334d14f4ba936be647beae06171f174a1890f747ab201bd25ccd53a37b73"}
Sep 18 13:38:38.555648 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:38.555601    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq" podStartSLOduration=2.555587605 podStartE2EDuration="2.555587605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:38:38.55355875 +0000 UTC m=+1415.909018575" watchObservedRunningTime="2026-09-18 13:38:38.555587605 +0000 UTC m=+1415.911047430"
Sep 18 13:38:47.293296 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:47.293209    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:38:47.293296 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:47.293268    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:38:47.307531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:47.307504    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:38:47.598655 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:47.598581    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:38:53.167253 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:38:53.167216    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:39:30.019448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:30.019412    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:39:34.152480 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:34.152443    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj"]
Sep 18 13:39:34.152922 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:34.152668    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj" podUID="386d6a34-3298-4906-a11e-ab7c6bd43aef" containerName="payload-processing" containerID="cri-o://ae81539d36a68c65342755de017eaec3ff2967aabee459fdf1ec7011926dc5b9" gracePeriod=30
Sep 18 13:39:34.423417 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:34.423353    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj"
Sep 18 13:39:34.475819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:34.475789    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386d6a34-3298-4906-a11e-ab7c6bd43aef-plugins-config-volume\") pod \"386d6a34-3298-4906-a11e-ab7c6bd43aef\" (UID: \"386d6a34-3298-4906-a11e-ab7c6bd43aef\") "
Sep 18 13:39:34.475998 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:34.475964    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhwz7\" (UniqueName: \"kubernetes.io/projected/386d6a34-3298-4906-a11e-ab7c6bd43aef-kube-api-access-vhwz7\") pod \"386d6a34-3298-4906-a11e-ab7c6bd43aef\" (UID: \"386d6a34-3298-4906-a11e-ab7c6bd43aef\") "
Sep 18 13:39:34.476264 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:34.476240    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386d6a34-3298-4906-a11e-ab7c6bd43aef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "386d6a34-3298-4906-a11e-ab7c6bd43aef" (UID: "386d6a34-3298-4906-a11e-ab7c6bd43aef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:39:34.476327 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:34.476314    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/386d6a34-3298-4906-a11e-ab7c6bd43aef-plugins-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:39:34.478212 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:34.478189    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386d6a34-3298-4906-a11e-ab7c6bd43aef-kube-api-access-vhwz7" (OuterVolumeSpecName: "kube-api-access-vhwz7") pod "386d6a34-3298-4906-a11e-ab7c6bd43aef" (UID: "386d6a34-3298-4906-a11e-ab7c6bd43aef"). InnerVolumeSpecName "kube-api-access-vhwz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:39:34.577236 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:34.577201    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhwz7\" (UniqueName: \"kubernetes.io/projected/386d6a34-3298-4906-a11e-ab7c6bd43aef-kube-api-access-vhwz7\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:39:34.851509 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:34.851434    2586 generic.go:356] "Generic (PLEG): container finished" podID="386d6a34-3298-4906-a11e-ab7c6bd43aef" containerID="ae81539d36a68c65342755de017eaec3ff2967aabee459fdf1ec7011926dc5b9" exitCode=0
Sep 18 13:39:34.851509 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:34.851472    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj"
Sep 18 13:39:34.851694 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:34.851510    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj" event={"ID":"386d6a34-3298-4906-a11e-ab7c6bd43aef","Type":"ContainerDied","Data":"ae81539d36a68c65342755de017eaec3ff2967aabee459fdf1ec7011926dc5b9"}
Sep 18 13:39:34.851694 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:34.851546    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj" event={"ID":"386d6a34-3298-4906-a11e-ab7c6bd43aef","Type":"ContainerDied","Data":"a950e9adbc6c0939cd943826bb8429947ec98e80a4dcd0dc4b9b0aa81fca6bc6"}
Sep 18 13:39:34.851694 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:34.851566    2586 scope.go:117] "RemoveContainer" containerID="ae81539d36a68c65342755de017eaec3ff2967aabee459fdf1ec7011926dc5b9"
Sep 18 13:39:34.861356 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:34.861331    2586 scope.go:117] "RemoveContainer" containerID="ae81539d36a68c65342755de017eaec3ff2967aabee459fdf1ec7011926dc5b9"
Sep 18 13:39:34.861649 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:39:34.861628    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae81539d36a68c65342755de017eaec3ff2967aabee459fdf1ec7011926dc5b9\": container with ID starting with ae81539d36a68c65342755de017eaec3ff2967aabee459fdf1ec7011926dc5b9 not found: ID does not exist" containerID="ae81539d36a68c65342755de017eaec3ff2967aabee459fdf1ec7011926dc5b9"
Sep 18 13:39:34.861700 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:34.861660    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae81539d36a68c65342755de017eaec3ff2967aabee459fdf1ec7011926dc5b9"} err="failed to get container status \"ae81539d36a68c65342755de017eaec3ff2967aabee459fdf1ec7011926dc5b9\": rpc error: code = NotFound desc = could not find container \"ae81539d36a68c65342755de017eaec3ff2967aabee459fdf1ec7011926dc5b9\": container with ID starting with ae81539d36a68c65342755de017eaec3ff2967aabee459fdf1ec7011926dc5b9 not found: ID does not exist"
Sep 18 13:39:34.882215 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:34.882181    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj"]
Sep 18 13:39:34.885683 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:34.885660    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-fa9905fd-596666c4b5-hdjpj"]
Sep 18 13:39:36.849826 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:36.849790    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="386d6a34-3298-4906-a11e-ab7c6bd43aef" path="/var/lib/kubelet/pods/386d6a34-3298-4906-a11e-ab7c6bd43aef/volumes"
Sep 18 13:39:43.147841 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:43.147798    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"]
Sep 18 13:39:58.167574 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.167508    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd" podUID="777420fd-638d-4703-934d-5b31b361c67a" containerName="main" containerID="cri-o://27568dc7195263fbd5e0dbfc9728b55c63ec5ab9b3f9f8678d03b84b91cb4892" gracePeriod=45
Sep 18 13:39:58.419637 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.419575    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:39:58.520765 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.520731    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5vtd\" (UniqueName: \"kubernetes.io/projected/777420fd-638d-4703-934d-5b31b361c67a-kube-api-access-m5vtd\") pod \"777420fd-638d-4703-934d-5b31b361c67a\" (UID: \"777420fd-638d-4703-934d-5b31b361c67a\") "
Sep 18 13:39:58.520974 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.520796    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/777420fd-638d-4703-934d-5b31b361c67a-home\") pod \"777420fd-638d-4703-934d-5b31b361c67a\" (UID: \"777420fd-638d-4703-934d-5b31b361c67a\") "
Sep 18 13:39:58.520974 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.520820    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/777420fd-638d-4703-934d-5b31b361c67a-tmp-dir\") pod \"777420fd-638d-4703-934d-5b31b361c67a\" (UID: \"777420fd-638d-4703-934d-5b31b361c67a\") "
Sep 18 13:39:58.520974 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.520837    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/777420fd-638d-4703-934d-5b31b361c67a-model-cache\") pod \"777420fd-638d-4703-934d-5b31b361c67a\" (UID: \"777420fd-638d-4703-934d-5b31b361c67a\") "
Sep 18 13:39:58.520974 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.520865    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/777420fd-638d-4703-934d-5b31b361c67a-dshm\") pod \"777420fd-638d-4703-934d-5b31b361c67a\" (UID: \"777420fd-638d-4703-934d-5b31b361c67a\") "
Sep 18 13:39:58.520974 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.520893    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/777420fd-638d-4703-934d-5b31b361c67a-tls-certs\") pod \"777420fd-638d-4703-934d-5b31b361c67a\" (UID: \"777420fd-638d-4703-934d-5b31b361c67a\") "
Sep 18 13:39:58.521236 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.521097    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777420fd-638d-4703-934d-5b31b361c67a-model-cache" (OuterVolumeSpecName: "model-cache") pod "777420fd-638d-4703-934d-5b31b361c67a" (UID: "777420fd-638d-4703-934d-5b31b361c67a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:39:58.521236 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.521117    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777420fd-638d-4703-934d-5b31b361c67a-home" (OuterVolumeSpecName: "home") pod "777420fd-638d-4703-934d-5b31b361c67a" (UID: "777420fd-638d-4703-934d-5b31b361c67a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:39:58.521236 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.521136    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777420fd-638d-4703-934d-5b31b361c67a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "777420fd-638d-4703-934d-5b31b361c67a" (UID: "777420fd-638d-4703-934d-5b31b361c67a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:39:58.521428 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.521411    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/777420fd-638d-4703-934d-5b31b361c67a-home\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:39:58.521483 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.521435    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/777420fd-638d-4703-934d-5b31b361c67a-tmp-dir\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:39:58.521483 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.521451    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/777420fd-638d-4703-934d-5b31b361c67a-model-cache\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:39:58.523193 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.523167    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777420fd-638d-4703-934d-5b31b361c67a-dshm" (OuterVolumeSpecName: "dshm") pod "777420fd-638d-4703-934d-5b31b361c67a" (UID: "777420fd-638d-4703-934d-5b31b361c67a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:39:58.523193 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.523173    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/777420fd-638d-4703-934d-5b31b361c67a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "777420fd-638d-4703-934d-5b31b361c67a" (UID: "777420fd-638d-4703-934d-5b31b361c67a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:39:58.523344 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.523201    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777420fd-638d-4703-934d-5b31b361c67a-kube-api-access-m5vtd" (OuterVolumeSpecName: "kube-api-access-m5vtd") pod "777420fd-638d-4703-934d-5b31b361c67a" (UID: "777420fd-638d-4703-934d-5b31b361c67a"). InnerVolumeSpecName "kube-api-access-m5vtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:39:58.622989 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.622956    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/777420fd-638d-4703-934d-5b31b361c67a-dshm\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:39:58.622989 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.622984    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/777420fd-638d-4703-934d-5b31b361c67a-tls-certs\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:39:58.622989 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.622995    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5vtd\" (UniqueName: \"kubernetes.io/projected/777420fd-638d-4703-934d-5b31b361c67a-kube-api-access-m5vtd\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:39:58.974218 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.974182    2586 generic.go:356] "Generic (PLEG): container finished" podID="777420fd-638d-4703-934d-5b31b361c67a" containerID="27568dc7195263fbd5e0dbfc9728b55c63ec5ab9b3f9f8678d03b84b91cb4892" exitCode=0
Sep 18 13:39:58.974392 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.974234    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"
Sep 18 13:39:58.974392 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.974285    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd" event={"ID":"777420fd-638d-4703-934d-5b31b361c67a","Type":"ContainerDied","Data":"27568dc7195263fbd5e0dbfc9728b55c63ec5ab9b3f9f8678d03b84b91cb4892"}
Sep 18 13:39:58.974392 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.974337    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd" event={"ID":"777420fd-638d-4703-934d-5b31b361c67a","Type":"ContainerDied","Data":"a7c809a6305ab97d685fabb34000455ec832cc311acf5a80cc93599f3ec25130"}
Sep 18 13:39:58.974392 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.974357    2586 scope.go:117] "RemoveContainer" containerID="27568dc7195263fbd5e0dbfc9728b55c63ec5ab9b3f9f8678d03b84b91cb4892"
Sep 18 13:39:58.983162 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.983144    2586 scope.go:117] "RemoveContainer" containerID="27568dc7195263fbd5e0dbfc9728b55c63ec5ab9b3f9f8678d03b84b91cb4892"
Sep 18 13:39:58.983424 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:39:58.983406    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27568dc7195263fbd5e0dbfc9728b55c63ec5ab9b3f9f8678d03b84b91cb4892\": container with ID starting with 27568dc7195263fbd5e0dbfc9728b55c63ec5ab9b3f9f8678d03b84b91cb4892 not found: ID does not exist" containerID="27568dc7195263fbd5e0dbfc9728b55c63ec5ab9b3f9f8678d03b84b91cb4892"
Sep 18 13:39:58.983489 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.983437    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27568dc7195263fbd5e0dbfc9728b55c63ec5ab9b3f9f8678d03b84b91cb4892"} err="failed to get container status \"27568dc7195263fbd5e0dbfc9728b55c63ec5ab9b3f9f8678d03b84b91cb4892\": rpc error: code = NotFound desc = could not find container \"27568dc7195263fbd5e0dbfc9728b55c63ec5ab9b3f9f8678d03b84b91cb4892\": container with ID starting with 27568dc7195263fbd5e0dbfc9728b55c63ec5ab9b3f9f8678d03b84b91cb4892 not found: ID does not exist"
Sep 18 13:39:58.993046 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.993020    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"]
Sep 18 13:39:58.996931 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:39:58.996891    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-fa9905fd/e2e-lifecycle-model-fa9905fd-kserve-8484f59775-f8tzd"]
Sep 18 13:40:00.850584 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:40:00.850550    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="777420fd-638d-4703-934d-5b31b361c67a" path="/var/lib/kubelet/pods/777420fd-638d-4703-934d-5b31b361c67a/volumes"
Sep 18 13:40:03.211561 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:40:03.211522    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:40:04.905635 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:40:04.905606    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lcqcw_1cc32441-1189-4113-bf17-2769fa23a747/ovn-acl-logging/0.log"
Sep 18 13:40:04.906121 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:40:04.905930    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lcqcw_1cc32441-1189-4113-bf17-2769fa23a747/ovn-acl-logging/0.log"
Sep 18 13:40:05.003592 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:40:05.003553    2586 scope.go:117] "RemoveContainer" containerID="e0a90d7335aef6213befa4f4206649ee18441226e42fdab2a7647f6acbf030da"
Sep 18 13:40:37.309609 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:40:37.309534    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:41:19.793088 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:19.793048    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:41:42.590317 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.590256    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"]
Sep 18 13:41:42.590822 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.590679    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="386d6a34-3298-4906-a11e-ab7c6bd43aef" containerName="payload-processing"
Sep 18 13:41:42.590822 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.590690    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="386d6a34-3298-4906-a11e-ab7c6bd43aef" containerName="payload-processing"
Sep 18 13:41:42.590822 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.590700    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="777420fd-638d-4703-934d-5b31b361c67a" containerName="main"
Sep 18 13:41:42.590822 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.590706    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="777420fd-638d-4703-934d-5b31b361c67a" containerName="main"
Sep 18 13:41:42.590822 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.590782    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="386d6a34-3298-4906-a11e-ab7c6bd43aef" containerName="payload-processing"
Sep 18 13:41:42.590822 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.590794    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="777420fd-638d-4703-934d-5b31b361c67a" containerName="main"
Sep 18 13:41:42.594016 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.593998    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.597040 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.597019    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-bc95f3-openshift-default-dockercfg-7pqnm\""
Sep 18 13:41:42.628875 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.628793    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"]
Sep 18 13:41:42.631878 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.631844    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-workload-socket\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.632032 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.631941    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istio-podinfo\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.632032 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.631969    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6vjl\" (UniqueName: \"kubernetes.io/projected/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-kube-api-access-z6vjl\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.632032 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.631995    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-credential-socket\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.632169 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.632058    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istio-data\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.632169 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.632110    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istio-envoy\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.632169 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.632133    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-workload-certs\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.632169 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.632158    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istiod-ca-cert\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.632314 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.632198    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istio-token\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.732789 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.732749    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istio-data\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.733047 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.732808    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istio-envoy\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.733047 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.732837    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-workload-certs\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.733047 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.732863    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istiod-ca-cert\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.733047 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.732949    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istio-token\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.733047 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.733012    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-workload-socket\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.733327 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.733053    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istio-podinfo\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.733327 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.733080    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6vjl\" (UniqueName: \"kubernetes.io/projected/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-kube-api-access-z6vjl\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.733327 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.733107    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-credential-socket\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.733327 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.733252    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istio-data\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.733542 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.733366    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-workload-certs\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.733591 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.733535    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-credential-socket\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.733676 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.733651    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-workload-socket\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.733721 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.733655    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istiod-ca-cert\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.735472 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.735451    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istio-envoy\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.735671 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.735653    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istio-podinfo\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.743395 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.743367    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istio-token\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.743605 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.743567    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6vjl\" (UniqueName: \"kubernetes.io/projected/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-kube-api-access-z6vjl\") pod \"e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") " pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:42.904872 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:42.904810    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:43.215395 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:43.215364    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"]
Sep 18 13:41:43.217377 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:41:43.217345    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacba9062_9a99_4e5c_ae7a_0adf44ce6b72.slice/crio-a6f963a4bfd753683eab2d97874ab349ef740186f679ded6b0570182ffe33d9b WatchSource:0}: Error finding container a6f963a4bfd753683eab2d97874ab349ef740186f679ded6b0570182ffe33d9b: Status 404 returned error can't find the container with id a6f963a4bfd753683eab2d97874ab349ef740186f679ded6b0570182ffe33d9b
Sep 18 13:41:43.219943 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:43.219888    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:41:43.220037 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:43.219978    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:41:43.220037 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:43.220008    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:41:43.512018 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:43.511930    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2" event={"ID":"acba9062-9a99-4e5c-ae7a-0adf44ce6b72","Type":"ContainerStarted","Data":"8ec938f0d5e0bc61a4ed5b7ca03e2e9474a5ace0f7c123c8cb8e6d2f9c131172"}
Sep 18 13:41:43.512018 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:43.511989    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2" event={"ID":"acba9062-9a99-4e5c-ae7a-0adf44ce6b72","Type":"ContainerStarted","Data":"a6f963a4bfd753683eab2d97874ab349ef740186f679ded6b0570182ffe33d9b"}
Sep 18 13:41:43.534995 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:43.534897    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2" podStartSLOduration=1.5348793619999999 podStartE2EDuration="1.534879362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:41:43.532484052 +0000 UTC m=+1600.887943887" watchObservedRunningTime="2026-09-18 13:41:43.534879362 +0000 UTC m=+1600.890339188"
Sep 18 13:41:43.905421 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:43.905381    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:44.911220 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:44.911138    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:45.521839 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:45.521803    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:45.522766 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:45.522746    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:41:52.724155 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:52.724123    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx"]
Sep 18 13:41:52.728166 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:52.728149    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx"
Sep 18 13:41:52.731662 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:52.731635    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-bc95f3\""
Sep 18 13:41:52.731785 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:52.731755    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-bc95f3-dockercfg-zdn65\""
Sep 18 13:41:52.742826 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:52.742802    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx"]
Sep 18 13:41:52.831573 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:52.831537    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe78327d-7236-4ad7-b3c7-709d7c17d32f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx\" (UID: \"fe78327d-7236-4ad7-b3c7-709d7c17d32f\") " pod="openshift-ingress/payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx"
Sep 18 13:41:52.831573 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:52.831576    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76vsn\" (UniqueName: \"kubernetes.io/projected/fe78327d-7236-4ad7-b3c7-709d7c17d32f-kube-api-access-76vsn\") pod \"payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx\" (UID: \"fe78327d-7236-4ad7-b3c7-709d7c17d32f\") " pod="openshift-ingress/payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx"
Sep 18 13:41:52.932371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:52.932332    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe78327d-7236-4ad7-b3c7-709d7c17d32f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx\" (UID: \"fe78327d-7236-4ad7-b3c7-709d7c17d32f\") " pod="openshift-ingress/payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx"
Sep 18 13:41:52.932371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:52.932378    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76vsn\" (UniqueName: \"kubernetes.io/projected/fe78327d-7236-4ad7-b3c7-709d7c17d32f-kube-api-access-76vsn\") pod \"payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx\" (UID: \"fe78327d-7236-4ad7-b3c7-709d7c17d32f\") " pod="openshift-ingress/payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx"
Sep 18 13:41:52.933059 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:52.933038    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe78327d-7236-4ad7-b3c7-709d7c17d32f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx\" (UID: \"fe78327d-7236-4ad7-b3c7-709d7c17d32f\") " pod="openshift-ingress/payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx"
Sep 18 13:41:52.940610 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:52.940569    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76vsn\" (UniqueName: \"kubernetes.io/projected/fe78327d-7236-4ad7-b3c7-709d7c17d32f-kube-api-access-76vsn\") pod \"payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx\" (UID: \"fe78327d-7236-4ad7-b3c7-709d7c17d32f\") " pod="openshift-ingress/payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx"
Sep 18 13:41:53.038688 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:53.038610    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx"
Sep 18 13:41:53.317891 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:53.317818    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx"]
Sep 18 13:41:53.320752 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:41:53.320727    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe78327d_7236_4ad7_b3c7_709d7c17d32f.slice/crio-ee5543e4a03d51b2d7d8d200a9b46c4784a0a603c7a69157f7b34df7713508f1 WatchSource:0}: Error finding container ee5543e4a03d51b2d7d8d200a9b46c4784a0a603c7a69157f7b34df7713508f1: Status 404 returned error can't find the container with id ee5543e4a03d51b2d7d8d200a9b46c4784a0a603c7a69157f7b34df7713508f1
Sep 18 13:41:53.566023 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:53.565983    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx" event={"ID":"fe78327d-7236-4ad7-b3c7-709d7c17d32f","Type":"ContainerStarted","Data":"6bdbb4929fdcbb6ee58250a5425f80d0201136721e04354a127502715d3b2be3"}
Sep 18 13:41:53.566201 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:53.566048    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx" event={"ID":"fe78327d-7236-4ad7-b3c7-709d7c17d32f","Type":"ContainerStarted","Data":"ee5543e4a03d51b2d7d8d200a9b46c4784a0a603c7a69157f7b34df7713508f1"}
Sep 18 13:41:53.566201 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:53.566143    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx"
Sep 18 13:41:53.585752 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:53.585661    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx" podStartSLOduration=1.585647509 podStartE2EDuration="1.585647509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:41:53.584366798 +0000 UTC m=+1610.939826623" watchObservedRunningTime="2026-09-18 13:41:53.585647509 +0000 UTC m=+1610.941107335"
Sep 18 13:41:54.979064 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:41:54.979032    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:42:04.573362 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:04.573324    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx"
Sep 18 13:42:16.709805 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.709641    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"]
Sep 18 13:42:16.714753 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.714732    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.719727 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.719708    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-57732b-openshift-default-dockercfg-qp4hj\""
Sep 18 13:42:16.735125 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.735098    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"]
Sep 18 13:42:16.861784 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.861754    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr7zx\" (UniqueName: \"kubernetes.io/projected/6de126ef-158b-46a2-ae50-8e664270679d-kube-api-access-dr7zx\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.861980 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.861829    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6de126ef-158b-46a2-ae50-8e664270679d-istiod-ca-cert\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.861980 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.861866    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-workload-certs\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.861980 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.861889    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-workload-socket\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.862174 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.862020    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6de126ef-158b-46a2-ae50-8e664270679d-istio-token\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.862174 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.862082    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-istio-data\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.862174 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.862114    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-credential-socket\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.862174 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.862141    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-istio-envoy\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.862174 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.862165    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6de126ef-158b-46a2-ae50-8e664270679d-istio-podinfo\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.962808 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.962715    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-workload-certs\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.962808 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.962756    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-workload-socket\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.962808 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.962795    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6de126ef-158b-46a2-ae50-8e664270679d-istio-token\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.963145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.962835    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-istio-data\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.963145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.962863    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-credential-socket\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.963145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.962885    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-istio-envoy\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.963145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.962942    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6de126ef-158b-46a2-ae50-8e664270679d-istio-podinfo\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.963145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.962981    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr7zx\" (UniqueName: \"kubernetes.io/projected/6de126ef-158b-46a2-ae50-8e664270679d-kube-api-access-dr7zx\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.963145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.963105    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6de126ef-158b-46a2-ae50-8e664270679d-istiod-ca-cert\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.963437 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.963178    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-workload-certs\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.963437 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.963216    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-workload-socket\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.963437 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.963399    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-istio-data\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.963601 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.963538    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-credential-socket\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.963881 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.963864    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6de126ef-158b-46a2-ae50-8e664270679d-istiod-ca-cert\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.965487 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.965464    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-istio-envoy\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.965633 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.965616    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6de126ef-158b-46a2-ae50-8e664270679d-istio-podinfo\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.972921 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.972884    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6de126ef-158b-46a2-ae50-8e664270679d-istio-token\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:16.974973 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:16.974950    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr7zx\" (UniqueName: \"kubernetes.io/projected/6de126ef-158b-46a2-ae50-8e664270679d-kube-api-access-dr7zx\") pod \"e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") " pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:17.025203 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:17.025168    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:17.324779 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:17.324748    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"]
Sep 18 13:42:17.326790 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:42:17.326760    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6de126ef_158b_46a2_ae50_8e664270679d.slice/crio-dbb9920cc04af7b58b7cbd27a0e46346b019612119601c7b43b5c2fc69818772 WatchSource:0}: Error finding container dbb9920cc04af7b58b7cbd27a0e46346b019612119601c7b43b5c2fc69818772: Status 404 returned error can't find the container with id dbb9920cc04af7b58b7cbd27a0e46346b019612119601c7b43b5c2fc69818772
Sep 18 13:42:17.328967 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:17.328936    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:42:17.329077 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:17.329015    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:42:17.329077 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:17.329059    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:42:17.696954 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:17.696890    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z" event={"ID":"6de126ef-158b-46a2-ae50-8e664270679d","Type":"ContainerStarted","Data":"2197ee1f06e795c0a342c2780e122dbc0745c504da97cb5b4ee1a91880602952"}
Sep 18 13:42:17.697158 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:17.696977    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z" event={"ID":"6de126ef-158b-46a2-ae50-8e664270679d","Type":"ContainerStarted","Data":"dbb9920cc04af7b58b7cbd27a0e46346b019612119601c7b43b5c2fc69818772"}
Sep 18 13:42:17.723406 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:17.723346    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z" podStartSLOduration=1.7233281630000001 podStartE2EDuration="1.723328163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:42:17.722346644 +0000 UTC m=+1635.077806470" watchObservedRunningTime="2026-09-18 13:42:17.723328163 +0000 UTC m=+1635.078787988"
Sep 18 13:42:18.025553 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:18.025455    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:18.027339 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:18.027303    2586 patch_prober.go:28] interesting pod/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 18 13:42:18.027507 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:18.027369    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z" podUID="6de126ef-158b-46a2-ae50-8e664270679d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 18 13:42:19.030458 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:19.030421    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:19.710955 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:19.710887    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:19.711851 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:19.711827    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:42:31.564104 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:31.564060    2586 patch_prober.go:28] interesting pod/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 18 13:42:31.564492 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:31.564129    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2" podUID="acba9062-9a99-4e5c-ae7a-0adf44ce6b72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 13:42:33.789036 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:33.789003    2586 generic.go:356] "Generic (PLEG): container finished" podID="acba9062-9a99-4e5c-ae7a-0adf44ce6b72" containerID="8ec938f0d5e0bc61a4ed5b7ca03e2e9474a5ace0f7c123c8cb8e6d2f9c131172" exitCode=0
Sep 18 13:42:33.789471 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:33.789077    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2" event={"ID":"acba9062-9a99-4e5c-ae7a-0adf44ce6b72","Type":"ContainerDied","Data":"8ec938f0d5e0bc61a4ed5b7ca03e2e9474a5ace0f7c123c8cb8e6d2f9c131172"}
Sep 18 13:42:33.789516 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:33.789495    2586 scope.go:117] "RemoveContainer" containerID="8ec938f0d5e0bc61a4ed5b7ca03e2e9474a5ace0f7c123c8cb8e6d2f9c131172"
Sep 18 13:42:33.790746 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:33.790719    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:42:33.790839 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:33.790783    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:42:33.790839 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:33.790812    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:42:34.796236 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:34.796203    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2" event={"ID":"acba9062-9a99-4e5c-ae7a-0adf44ce6b72","Type":"ContainerStarted","Data":"6dfcac2386fb55bcbd3a527f957d2fee5f98c889c49e65430df5c16089785f40"}
Sep 18 13:42:34.909674 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:34.909273    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:42:34.909674 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:34.909385    2586 patch_prober.go:28] interesting pod/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 18 13:42:34.909674 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:34.909428    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2" podUID="acba9062-9a99-4e5c-ae7a-0adf44ce6b72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 18 13:42:35.906293 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:35.906258    2586 patch_prober.go:28] interesting pod/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 18 13:42:35.906865 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:35.906833    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2" podUID="acba9062-9a99-4e5c-ae7a-0adf44ce6b72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 18 13:42:36.906009 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:36.905972    2586 patch_prober.go:28] interesting pod/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 18 13:42:36.906221 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:36.906041    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2" podUID="acba9062-9a99-4e5c-ae7a-0adf44ce6b72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 18 13:42:37.906084 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:37.906046    2586 patch_prober.go:28] interesting pod/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 18 13:42:37.906677 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:37.906647    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2" podUID="acba9062-9a99-4e5c-ae7a-0adf44ce6b72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 18 13:42:38.906301 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:38.906255    2586 patch_prober.go:28] interesting pod/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 18 13:42:38.906796 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:38.906328    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2" podUID="acba9062-9a99-4e5c-ae7a-0adf44ce6b72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 18 13:42:39.908848 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:39.908329    2586 patch_prober.go:28] interesting pod/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 18 13:42:39.908848 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:39.908393    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2" podUID="acba9062-9a99-4e5c-ae7a-0adf44ce6b72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 18 13:42:40.909255 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:40.909212    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:42:40.909635 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:40.909613    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:42:40.910321 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:40.910299    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:42:49.366346 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:49.366303    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:42:59.949232 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:59.949192    2586 generic.go:356] "Generic (PLEG): container finished" podID="6de126ef-158b-46a2-ae50-8e664270679d" containerID="2197ee1f06e795c0a342c2780e122dbc0745c504da97cb5b4ee1a91880602952" exitCode=0
Sep 18 13:42:59.949874 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:59.949274    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z" event={"ID":"6de126ef-158b-46a2-ae50-8e664270679d","Type":"ContainerDied","Data":"2197ee1f06e795c0a342c2780e122dbc0745c504da97cb5b4ee1a91880602952"}
Sep 18 13:42:59.949874 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:59.949690    2586 scope.go:117] "RemoveContainer" containerID="2197ee1f06e795c0a342c2780e122dbc0745c504da97cb5b4ee1a91880602952"
Sep 18 13:42:59.950878 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:59.950847    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:42:59.951024 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:59.950953    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:42:59.951024 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:42:59.951001    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:43:00.956886 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:00.956855    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z" event={"ID":"6de126ef-158b-46a2-ae50-8e664270679d","Type":"ContainerStarted","Data":"a9be7ab35103fdb18b34ed8d39b04c409a6b13d4ad174f4279831cbe4e9ec17c"}
Sep 18 13:43:01.026682 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:01.026638    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:43:01.028001 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:01.027961    2586 patch_prober.go:28] interesting pod/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 18 13:43:01.028143 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:01.028050    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z" podUID="6de126ef-158b-46a2-ae50-8e664270679d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 18 13:43:02.026401 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:02.026355    2586 patch_prober.go:28] interesting pod/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 18 13:43:02.027021 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:02.026431    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z" podUID="6de126ef-158b-46a2-ae50-8e664270679d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 18 13:43:03.025557 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:03.025511    2586 patch_prober.go:28] interesting pod/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 18 13:43:03.025762 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:03.025578    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z" podUID="6de126ef-158b-46a2-ae50-8e664270679d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 18 13:43:04.026406 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:04.026363    2586 patch_prober.go:28] interesting pod/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 18 13:43:04.027018 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:04.026983    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z" podUID="6de126ef-158b-46a2-ae50-8e664270679d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 18 13:43:04.711849 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:04.711809    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:43:05.026428 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:05.026328    2586 patch_prober.go:28] interesting pod/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 18 13:43:05.026880 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:05.026427    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z" podUID="6de126ef-158b-46a2-ae50-8e664270679d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 18 13:43:05.046293 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:05.046270    2586 scope.go:117] "RemoveContainer" containerID="94a4f49c5b006d8146121d4e3b0fc95505e508eba3c45473f1f60881c7ad4a3b"
Sep 18 13:43:06.026410 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:06.026379    2586 patch_prober.go:28] interesting pod/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Sep 18 13:43:06.026592 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:06.026444    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z" podUID="6de126ef-158b-46a2-ae50-8e664270679d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Sep 18 13:43:07.029633 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:07.029600    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:43:07.030830 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:07.030801    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:43:10.005472 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:10.005383    2586 generic.go:356] "Generic (PLEG): container finished" podID="ad7f15d2-7e4f-422c-816a-d35e644d573b" containerID="c68f2889f1bfbb70d939fc952f178820f9af01e4f41124a80e84ed7f5aee0b07" exitCode=137
Sep 18 13:43:10.005472 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:10.005459    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg" event={"ID":"ad7f15d2-7e4f-422c-816a-d35e644d573b","Type":"ContainerDied","Data":"c68f2889f1bfbb70d939fc952f178820f9af01e4f41124a80e84ed7f5aee0b07"}
Sep 18 13:43:10.005883 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:10.005830    2586 scope.go:117] "RemoveContainer" containerID="c68f2889f1bfbb70d939fc952f178820f9af01e4f41124a80e84ed7f5aee0b07"
Sep 18 13:43:11.013353 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:11.013323    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg" event={"ID":"ad7f15d2-7e4f-422c-816a-d35e644d573b","Type":"ContainerStarted","Data":"1b778bdc9b39a0494783175dadcc06943c4af77ad749a0e26cae9d90efbac30d"}
Sep 18 13:43:11.013720 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:11.013518    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg"
Sep 18 13:43:13.908758 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:13.908722    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx"]
Sep 18 13:43:13.909274 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:13.909023    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx" podUID="fe78327d-7236-4ad7-b3c7-709d7c17d32f" containerName="payload-processing" containerID="cri-o://6bdbb4929fdcbb6ee58250a5425f80d0201136721e04354a127502715d3b2be3" gracePeriod=30
Sep 18 13:43:14.030838 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:14.030808    2586 generic.go:356] "Generic (PLEG): container finished" podID="fe78327d-7236-4ad7-b3c7-709d7c17d32f" containerID="6bdbb4929fdcbb6ee58250a5425f80d0201136721e04354a127502715d3b2be3" exitCode=0
Sep 18 13:43:14.030992 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:14.030886    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx" event={"ID":"fe78327d-7236-4ad7-b3c7-709d7c17d32f","Type":"ContainerDied","Data":"6bdbb4929fdcbb6ee58250a5425f80d0201136721e04354a127502715d3b2be3"}
Sep 18 13:43:14.170407 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:14.170344    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx"
Sep 18 13:43:14.261267 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:14.261231    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76vsn\" (UniqueName: \"kubernetes.io/projected/fe78327d-7236-4ad7-b3c7-709d7c17d32f-kube-api-access-76vsn\") pod \"fe78327d-7236-4ad7-b3c7-709d7c17d32f\" (UID: \"fe78327d-7236-4ad7-b3c7-709d7c17d32f\") "
Sep 18 13:43:14.261417 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:14.261282    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe78327d-7236-4ad7-b3c7-709d7c17d32f-plugins-config-volume\") pod \"fe78327d-7236-4ad7-b3c7-709d7c17d32f\" (UID: \"fe78327d-7236-4ad7-b3c7-709d7c17d32f\") "
Sep 18 13:43:14.261695 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:14.261668    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe78327d-7236-4ad7-b3c7-709d7c17d32f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fe78327d-7236-4ad7-b3c7-709d7c17d32f" (UID: "fe78327d-7236-4ad7-b3c7-709d7c17d32f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:43:14.263525 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:14.263503    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe78327d-7236-4ad7-b3c7-709d7c17d32f-kube-api-access-76vsn" (OuterVolumeSpecName: "kube-api-access-76vsn") pod "fe78327d-7236-4ad7-b3c7-709d7c17d32f" (UID: "fe78327d-7236-4ad7-b3c7-709d7c17d32f"). InnerVolumeSpecName "kube-api-access-76vsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:43:14.362945 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:14.362887    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76vsn\" (UniqueName: \"kubernetes.io/projected/fe78327d-7236-4ad7-b3c7-709d7c17d32f-kube-api-access-76vsn\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:43:14.362945 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:14.362939    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe78327d-7236-4ad7-b3c7-709d7c17d32f-plugins-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:43:15.037143 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:15.037115    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx"
Sep 18 13:43:15.037143 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:15.037120    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx" event={"ID":"fe78327d-7236-4ad7-b3c7-709d7c17d32f","Type":"ContainerDied","Data":"ee5543e4a03d51b2d7d8d200a9b46c4784a0a603c7a69157f7b34df7713508f1"}
Sep 18 13:43:15.037648 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:15.037182    2586 scope.go:117] "RemoveContainer" containerID="6bdbb4929fdcbb6ee58250a5425f80d0201136721e04354a127502715d3b2be3"
Sep 18 13:43:15.064556 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:15.064532    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx"]
Sep 18 13:43:15.086800 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:15.086773    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bc95f3-77c4556c55-lkdrx"]
Sep 18 13:43:16.850604 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:16.850575    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe78327d-7236-4ad7-b3c7-709d7c17d32f" path="/var/lib/kubelet/pods/fe78327d-7236-4ad7-b3c7-709d7c17d32f/volumes"
Sep 18 13:43:22.021677 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:22.021641    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg"
Sep 18 13:43:29.493567 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:29.493534    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"]
Sep 18 13:43:29.494025 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:29.493734    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2" podUID="acba9062-9a99-4e5c-ae7a-0adf44ce6b72" containerName="istio-proxy" containerID="cri-o://6dfcac2386fb55bcbd3a527f957d2fee5f98c889c49e65430df5c16089785f40" gracePeriod=30
Sep 18 13:43:29.886805 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:29.886774    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57732b-xmb8m"]
Sep 18 13:43:29.887218 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:29.887204    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe78327d-7236-4ad7-b3c7-709d7c17d32f" containerName="payload-processing"
Sep 18 13:43:29.887284 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:29.887220    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe78327d-7236-4ad7-b3c7-709d7c17d32f" containerName="payload-processing"
Sep 18 13:43:29.887325 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:29.887301    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe78327d-7236-4ad7-b3c7-709d7c17d32f" containerName="payload-processing"
Sep 18 13:43:29.890374 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:29.890359    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57732b-xmb8m"
Sep 18 13:43:29.894238 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:29.894208    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nh2cd\""
Sep 18 13:43:29.894386 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:29.894293    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:43:29.914465 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:29.914435    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea329206-3942-4293-9e91-e6256c1620d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-57732b-xmb8m\" (UID: \"ea329206-3942-4293-9e91-e6256c1620d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57732b-xmb8m"
Sep 18 13:43:29.921433 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:29.921403    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57732b-xmb8m"]
Sep 18 13:43:30.015510 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:30.015479    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea329206-3942-4293-9e91-e6256c1620d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-57732b-xmb8m\" (UID: \"ea329206-3942-4293-9e91-e6256c1620d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57732b-xmb8m"
Sep 18 13:43:30.016297 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:30.016276    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea329206-3942-4293-9e91-e6256c1620d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-57732b-xmb8m\" (UID: \"ea329206-3942-4293-9e91-e6256c1620d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57732b-xmb8m"
Sep 18 13:43:30.200935 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:30.200830    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57732b-xmb8m"
Sep 18 13:43:30.500430 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:30.500350    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57732b-xmb8m"]
Sep 18 13:43:30.503880 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:43:30.503851    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea329206_3942_4293_9e91_e6256c1620d8.slice/crio-1eabff7a7430e880eeeb0c220895b42d8bf469631dddef7920be1e128aeea0e5 WatchSource:0}: Error finding container 1eabff7a7430e880eeeb0c220895b42d8bf469631dddef7920be1e128aeea0e5: Status 404 returned error can't find the container with id 1eabff7a7430e880eeeb0c220895b42d8bf469631dddef7920be1e128aeea0e5
Sep 18 13:43:31.132177 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:31.132086    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57732b-xmb8m" event={"ID":"ea329206-3942-4293-9e91-e6256c1620d8","Type":"ContainerStarted","Data":"dbaba5c5c32711b432d5ddd0e14079444e2197d6947272649da4a3c9bb785baa"}
Sep 18 13:43:31.132177 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:31.132141    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57732b-xmb8m" event={"ID":"ea329206-3942-4293-9e91-e6256c1620d8","Type":"ContainerStarted","Data":"1eabff7a7430e880eeeb0c220895b42d8bf469631dddef7920be1e128aeea0e5"}
Sep 18 13:43:31.176493 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:31.176441    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57732b-xmb8m" podStartSLOduration=2.176425246 podStartE2EDuration="2.176425246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:43:31.174988356 +0000 UTC m=+1708.530448173" watchObservedRunningTime="2026-09-18 13:43:31.176425246 +0000 UTC m=+1708.531885071"
Sep 18 13:43:32.138875 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:32.138840    2586 generic.go:356] "Generic (PLEG): container finished" podID="ea329206-3942-4293-9e91-e6256c1620d8" containerID="dbaba5c5c32711b432d5ddd0e14079444e2197d6947272649da4a3c9bb785baa" exitCode=0
Sep 18 13:43:32.139232 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:32.138898    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57732b-xmb8m" event={"ID":"ea329206-3942-4293-9e91-e6256c1620d8","Type":"ContainerDied","Data":"dbaba5c5c32711b432d5ddd0e14079444e2197d6947272649da4a3c9bb785baa"}
Sep 18 13:43:33.272769 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:33.272748    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57732b-xmb8m"
Sep 18 13:43:33.343408 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:33.343379    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea329206-3942-4293-9e91-e6256c1620d8-maas-api-service-ca\") pod \"ea329206-3942-4293-9e91-e6256c1620d8\" (UID: \"ea329206-3942-4293-9e91-e6256c1620d8\") "
Sep 18 13:43:33.343815 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:33.343786    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea329206-3942-4293-9e91-e6256c1620d8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ea329206-3942-4293-9e91-e6256c1620d8" (UID: "ea329206-3942-4293-9e91-e6256c1620d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:43:33.444213 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:33.444138    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea329206-3942-4293-9e91-e6256c1620d8-maas-api-service-ca\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:43:34.151768 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.151741    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57732b-xmb8m"
Sep 18 13:43:34.151969 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.151739    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57732b-xmb8m" event={"ID":"ea329206-3942-4293-9e91-e6256c1620d8","Type":"ContainerDied","Data":"1eabff7a7430e880eeeb0c220895b42d8bf469631dddef7920be1e128aeea0e5"}
Sep 18 13:43:34.151969 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.151862    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1eabff7a7430e880eeeb0c220895b42d8bf469631dddef7920be1e128aeea0e5"
Sep 18 13:43:34.753606 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.753520    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:43:34.858619 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.858592    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istio-podinfo\") pod \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") "
Sep 18 13:43:34.858619 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.858624    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6vjl\" (UniqueName: \"kubernetes.io/projected/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-kube-api-access-z6vjl\") pod \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") "
Sep 18 13:43:34.858806 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.858643    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-workload-socket\") pod \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") "
Sep 18 13:43:34.858806 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.858678    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istio-envoy\") pod \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") "
Sep 18 13:43:34.858806 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.858725    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istio-token\") pod \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") "
Sep 18 13:43:34.858806 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.858782    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istiod-ca-cert\") pod \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") "
Sep 18 13:43:34.859027 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.858868    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-credential-socket\") pod \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") "
Sep 18 13:43:34.859027 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.858918    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istio-data\") pod \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") "
Sep 18 13:43:34.859027 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.858967    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-workload-certs\") pod \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\" (UID: \"acba9062-9a99-4e5c-ae7a-0adf44ce6b72\") "
Sep 18 13:43:34.859329 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.858966    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "acba9062-9a99-4e5c-ae7a-0adf44ce6b72" (UID: "acba9062-9a99-4e5c-ae7a-0adf44ce6b72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:43:34.859329 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.859139    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istio-data" (OuterVolumeSpecName: "istio-data") pod "acba9062-9a99-4e5c-ae7a-0adf44ce6b72" (UID: "acba9062-9a99-4e5c-ae7a-0adf44ce6b72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:43:34.859329 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.859288    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "acba9062-9a99-4e5c-ae7a-0adf44ce6b72" (UID: "acba9062-9a99-4e5c-ae7a-0adf44ce6b72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:43:34.859329 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.859298    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "acba9062-9a99-4e5c-ae7a-0adf44ce6b72" (UID: "acba9062-9a99-4e5c-ae7a-0adf44ce6b72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:43:34.859766 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.859671    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "acba9062-9a99-4e5c-ae7a-0adf44ce6b72" (UID: "acba9062-9a99-4e5c-ae7a-0adf44ce6b72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:43:34.861105 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.861077    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "acba9062-9a99-4e5c-ae7a-0adf44ce6b72" (UID: "acba9062-9a99-4e5c-ae7a-0adf44ce6b72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:43:34.861229 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.861201    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istio-token" (OuterVolumeSpecName: "istio-token") pod "acba9062-9a99-4e5c-ae7a-0adf44ce6b72" (UID: "acba9062-9a99-4e5c-ae7a-0adf44ce6b72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:43:34.861394 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.861364    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "acba9062-9a99-4e5c-ae7a-0adf44ce6b72" (UID: "acba9062-9a99-4e5c-ae7a-0adf44ce6b72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 13:43:34.861394 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.861370    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-kube-api-access-z6vjl" (OuterVolumeSpecName: "kube-api-access-z6vjl") pod "acba9062-9a99-4e5c-ae7a-0adf44ce6b72" (UID: "acba9062-9a99-4e5c-ae7a-0adf44ce6b72"). InnerVolumeSpecName "kube-api-access-z6vjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:43:34.960753 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.960725    2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-workload-certs\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:43:34.960753 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.960751    2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istio-podinfo\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:43:34.960955 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.960760    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6vjl\" (UniqueName: \"kubernetes.io/projected/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-kube-api-access-z6vjl\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:43:34.960955 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.960771    2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-workload-socket\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:43:34.960955 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.960780    2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istio-envoy\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:43:34.960955 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.960788    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istio-token\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:43:34.960955 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.960796    2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istiod-ca-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:43:34.960955 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.960803    2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-credential-socket\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:43:34.960955 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:34.960811    2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acba9062-9a99-4e5c-ae7a-0adf44ce6b72-istio-data\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:43:35.165136 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:35.165107    2586 generic.go:356] "Generic (PLEG): container finished" podID="acba9062-9a99-4e5c-ae7a-0adf44ce6b72" containerID="6dfcac2386fb55bcbd3a527f957d2fee5f98c889c49e65430df5c16089785f40" exitCode=0
Sep 18 13:43:35.165312 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:35.165176    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"
Sep 18 13:43:35.165312 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:35.165192    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2" event={"ID":"acba9062-9a99-4e5c-ae7a-0adf44ce6b72","Type":"ContainerDied","Data":"6dfcac2386fb55bcbd3a527f957d2fee5f98c889c49e65430df5c16089785f40"}
Sep 18 13:43:35.165312 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:35.165285    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2" event={"ID":"acba9062-9a99-4e5c-ae7a-0adf44ce6b72","Type":"ContainerDied","Data":"a6f963a4bfd753683eab2d97874ab349ef740186f679ded6b0570182ffe33d9b"}
Sep 18 13:43:35.165312 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:35.165308    2586 scope.go:117] "RemoveContainer" containerID="6dfcac2386fb55bcbd3a527f957d2fee5f98c889c49e65430df5c16089785f40"
Sep 18 13:43:35.174985 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:35.174968    2586 scope.go:117] "RemoveContainer" containerID="8ec938f0d5e0bc61a4ed5b7ca03e2e9474a5ace0f7c123c8cb8e6d2f9c131172"
Sep 18 13:43:35.183071 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:35.183056    2586 scope.go:117] "RemoveContainer" containerID="6dfcac2386fb55bcbd3a527f957d2fee5f98c889c49e65430df5c16089785f40"
Sep 18 13:43:35.183318 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:43:35.183299    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dfcac2386fb55bcbd3a527f957d2fee5f98c889c49e65430df5c16089785f40\": container with ID starting with 6dfcac2386fb55bcbd3a527f957d2fee5f98c889c49e65430df5c16089785f40 not found: ID does not exist" containerID="6dfcac2386fb55bcbd3a527f957d2fee5f98c889c49e65430df5c16089785f40"
Sep 18 13:43:35.183355 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:35.183331    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dfcac2386fb55bcbd3a527f957d2fee5f98c889c49e65430df5c16089785f40"} err="failed to get container status \"6dfcac2386fb55bcbd3a527f957d2fee5f98c889c49e65430df5c16089785f40\": rpc error: code = NotFound desc = could not find container \"6dfcac2386fb55bcbd3a527f957d2fee5f98c889c49e65430df5c16089785f40\": container with ID starting with 6dfcac2386fb55bcbd3a527f957d2fee5f98c889c49e65430df5c16089785f40 not found: ID does not exist"
Sep 18 13:43:35.183355 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:35.183352    2586 scope.go:117] "RemoveContainer" containerID="8ec938f0d5e0bc61a4ed5b7ca03e2e9474a5ace0f7c123c8cb8e6d2f9c131172"
Sep 18 13:43:35.183587 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:43:35.183571    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ec938f0d5e0bc61a4ed5b7ca03e2e9474a5ace0f7c123c8cb8e6d2f9c131172\": container with ID starting with 8ec938f0d5e0bc61a4ed5b7ca03e2e9474a5ace0f7c123c8cb8e6d2f9c131172 not found: ID does not exist" containerID="8ec938f0d5e0bc61a4ed5b7ca03e2e9474a5ace0f7c123c8cb8e6d2f9c131172"
Sep 18 13:43:35.183634 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:35.183592    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ec938f0d5e0bc61a4ed5b7ca03e2e9474a5ace0f7c123c8cb8e6d2f9c131172"} err="failed to get container status \"8ec938f0d5e0bc61a4ed5b7ca03e2e9474a5ace0f7c123c8cb8e6d2f9c131172\": rpc error: code = NotFound desc = could not find container \"8ec938f0d5e0bc61a4ed5b7ca03e2e9474a5ace0f7c123c8cb8e6d2f9c131172\": container with ID starting with 8ec938f0d5e0bc61a4ed5b7ca03e2e9474a5ace0f7c123c8cb8e6d2f9c131172 not found: ID does not exist"
Sep 18 13:43:35.205665 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:35.205643    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"]
Sep 18 13:43:35.216550 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:35.216526    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-bc95f3-openshift-default-7f855964bd-brkk2"]
Sep 18 13:43:36.850630 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:36.850597    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acba9062-9a99-4e5c-ae7a-0adf44ce6b72" path="/var/lib/kubelet/pods/acba9062-9a99-4e5c-ae7a-0adf44ce6b72/volumes"
Sep 18 13:43:41.165131 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:41.165096    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:43:47.869960 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:47.869925    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57732b-xmb8m"]
Sep 18 13:43:47.876052 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:47.876020    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-57732b-xmb8m"]
Sep 18 13:43:48.850680 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:48.850644    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea329206-3942-4293-9e91-e6256c1620d8" path="/var/lib/kubelet/pods/ea329206-3942-4293-9e91-e6256c1620d8/volumes"
Sep 18 13:43:53.823655 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:53.823619    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"]
Sep 18 13:43:59.880819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:59.880768    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"]
Sep 18 13:43:59.881272 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:43:59.881021    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z" podUID="6de126ef-158b-46a2-ae50-8e664270679d" containerName="istio-proxy" containerID="cri-o://a9be7ab35103fdb18b34ed8d39b04c409a6b13d4ad174f4279831cbe4e9ec17c" gracePeriod=30
Sep 18 13:44:00.270293 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:00.270256    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b2d5da-bf6pd"]
Sep 18 13:44:00.270785 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:00.270765    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acba9062-9a99-4e5c-ae7a-0adf44ce6b72" containerName="istio-proxy"
Sep 18 13:44:00.270880 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:00.270789    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="acba9062-9a99-4e5c-ae7a-0adf44ce6b72" containerName="istio-proxy"
Sep 18 13:44:00.270880 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:00.270800    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acba9062-9a99-4e5c-ae7a-0adf44ce6b72" containerName="istio-proxy"
Sep 18 13:44:00.270880 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:00.270808    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="acba9062-9a99-4e5c-ae7a-0adf44ce6b72" containerName="istio-proxy"
Sep 18 13:44:00.270880 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:00.270818    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea329206-3942-4293-9e91-e6256c1620d8" containerName="revoke-keys"
Sep 18 13:44:00.270880 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:00.270826    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea329206-3942-4293-9e91-e6256c1620d8" containerName="revoke-keys"
Sep 18 13:44:00.271184 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:00.270955    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="acba9062-9a99-4e5c-ae7a-0adf44ce6b72" containerName="istio-proxy"
Sep 18 13:44:00.271184 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:00.270971    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="acba9062-9a99-4e5c-ae7a-0adf44ce6b72" containerName="istio-proxy"
Sep 18 13:44:00.271184 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:00.270980    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea329206-3942-4293-9e91-e6256c1620d8" containerName="revoke-keys"
Sep 18 13:44:00.273810 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:00.273790    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b2d5da-bf6pd"
Sep 18 13:44:00.277323 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:00.277302    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:44:00.277514 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:00.277501    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nh2cd\""
Sep 18 13:44:00.322166 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:00.322140    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b2d5da-bf6pd"]
Sep 18 13:44:00.401582 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:00.401548    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cdc05c5-1d49-40c2-9e36-edf6ef89a945-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b2d5da-bf6pd\" (UID: \"8cdc05c5-1d49-40c2-9e36-edf6ef89a945\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b2d5da-bf6pd"
Sep 18 13:44:00.502428 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:00.502395    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cdc05c5-1d49-40c2-9e36-edf6ef89a945-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b2d5da-bf6pd\" (UID: \"8cdc05c5-1d49-40c2-9e36-edf6ef89a945\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b2d5da-bf6pd"
Sep 18 13:44:00.503001 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:00.502982    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cdc05c5-1d49-40c2-9e36-edf6ef89a945-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b2d5da-bf6pd\" (UID: \"8cdc05c5-1d49-40c2-9e36-edf6ef89a945\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b2d5da-bf6pd"
Sep 18 13:44:00.584514 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:00.584442    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b2d5da-bf6pd"
Sep 18 13:44:00.897098 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:44:00.897057    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cdc05c5_1d49_40c2_9e36_edf6ef89a945.slice/crio-5d7261cd91542d2ce47d0291fb7a57ccbe3d6457bc5812ffce11db577842b6f7 WatchSource:0}: Error finding container 5d7261cd91542d2ce47d0291fb7a57ccbe3d6457bc5812ffce11db577842b6f7: Status 404 returned error can't find the container with id 5d7261cd91542d2ce47d0291fb7a57ccbe3d6457bc5812ffce11db577842b6f7
Sep 18 13:44:00.898673 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:00.898597    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b2d5da-bf6pd"]
Sep 18 13:44:01.305969 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:01.305930    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b2d5da-bf6pd" event={"ID":"8cdc05c5-1d49-40c2-9e36-edf6ef89a945","Type":"ContainerStarted","Data":"0f4e53b6a18d433e9f81b4c3d5fe4612d04e3293b317ea219227ad7ff1a04fc8"}
Sep 18 13:44:01.306112 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:01.305995    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b2d5da-bf6pd" event={"ID":"8cdc05c5-1d49-40c2-9e36-edf6ef89a945","Type":"ContainerStarted","Data":"5d7261cd91542d2ce47d0291fb7a57ccbe3d6457bc5812ffce11db577842b6f7"}
Sep 18 13:44:01.356297 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:01.356250    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b2d5da-bf6pd" podStartSLOduration=1.356233268 podStartE2EDuration="1.356233268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:44:01.353795405 +0000 UTC m=+1738.709255231" watchObservedRunningTime="2026-09-18 13:44:01.356233268 +0000 UTC m=+1738.711693317"
Sep 18 13:44:02.314277 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:02.314244    2586 generic.go:356] "Generic (PLEG): container finished" podID="8cdc05c5-1d49-40c2-9e36-edf6ef89a945" containerID="0f4e53b6a18d433e9f81b4c3d5fe4612d04e3293b317ea219227ad7ff1a04fc8" exitCode=0
Sep 18 13:44:02.314648 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:02.314344    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b2d5da-bf6pd" event={"ID":"8cdc05c5-1d49-40c2-9e36-edf6ef89a945","Type":"ContainerDied","Data":"0f4e53b6a18d433e9f81b4c3d5fe4612d04e3293b317ea219227ad7ff1a04fc8"}
Sep 18 13:44:03.451680 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:03.451657    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b2d5da-bf6pd"
Sep 18 13:44:03.534813 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:03.534783    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cdc05c5-1d49-40c2-9e36-edf6ef89a945-maas-api-service-ca\") pod \"8cdc05c5-1d49-40c2-9e36-edf6ef89a945\" (UID: \"8cdc05c5-1d49-40c2-9e36-edf6ef89a945\") "
Sep 18 13:44:03.535131 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:03.535108    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cdc05c5-1d49-40c2-9e36-edf6ef89a945-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8cdc05c5-1d49-40c2-9e36-edf6ef89a945" (UID: "8cdc05c5-1d49-40c2-9e36-edf6ef89a945"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:44:03.636457 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:03.636432    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cdc05c5-1d49-40c2-9e36-edf6ef89a945-maas-api-service-ca\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:04.327433 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:04.327402    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b2d5da-bf6pd"
Sep 18 13:44:04.327433 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:04.327423    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b2d5da-bf6pd" event={"ID":"8cdc05c5-1d49-40c2-9e36-edf6ef89a945","Type":"ContainerDied","Data":"5d7261cd91542d2ce47d0291fb7a57ccbe3d6457bc5812ffce11db577842b6f7"}
Sep 18 13:44:04.327637 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:04.327476    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d7261cd91542d2ce47d0291fb7a57ccbe3d6457bc5812ffce11db577842b6f7"
Sep 18 13:44:05.147499 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.147428    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:44:05.252882 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.252847    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dr7zx\" (UniqueName: \"kubernetes.io/projected/6de126ef-158b-46a2-ae50-8e664270679d-kube-api-access-dr7zx\") pod \"6de126ef-158b-46a2-ae50-8e664270679d\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") "
Sep 18 13:44:05.253066 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.252889    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6de126ef-158b-46a2-ae50-8e664270679d-istio-token\") pod \"6de126ef-158b-46a2-ae50-8e664270679d\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") "
Sep 18 13:44:05.253066 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.252993    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-workload-certs\") pod \"6de126ef-158b-46a2-ae50-8e664270679d\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") "
Sep 18 13:44:05.253167 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.253142    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-workload-socket\") pod \"6de126ef-158b-46a2-ae50-8e664270679d\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") "
Sep 18 13:44:05.253245 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.253230    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6de126ef-158b-46a2-ae50-8e664270679d-istio-podinfo\") pod \"6de126ef-158b-46a2-ae50-8e664270679d\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") "
Sep 18 13:44:05.253315 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.253274    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6de126ef-158b-46a2-ae50-8e664270679d" (UID: "6de126ef-158b-46a2-ae50-8e664270679d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:05.253315 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.253280    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6de126ef-158b-46a2-ae50-8e664270679d-istiod-ca-cert\") pod \"6de126ef-158b-46a2-ae50-8e664270679d\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") "
Sep 18 13:44:05.253413 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.253379    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-credential-socket\") pod \"6de126ef-158b-46a2-ae50-8e664270679d\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") "
Sep 18 13:44:05.253413 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.253376    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6de126ef-158b-46a2-ae50-8e664270679d" (UID: "6de126ef-158b-46a2-ae50-8e664270679d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:05.253413 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.253410    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-istio-data\") pod \"6de126ef-158b-46a2-ae50-8e664270679d\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") "
Sep 18 13:44:05.253586 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.253470    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-istio-envoy\") pod \"6de126ef-158b-46a2-ae50-8e664270679d\" (UID: \"6de126ef-158b-46a2-ae50-8e664270679d\") "
Sep 18 13:44:05.253655 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.253582    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6de126ef-158b-46a2-ae50-8e664270679d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6de126ef-158b-46a2-ae50-8e664270679d" (UID: "6de126ef-158b-46a2-ae50-8e664270679d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:44:05.253655 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.253585    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6de126ef-158b-46a2-ae50-8e664270679d" (UID: "6de126ef-158b-46a2-ae50-8e664270679d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:05.253964 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.253934    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-istio-data" (OuterVolumeSpecName: "istio-data") pod "6de126ef-158b-46a2-ae50-8e664270679d" (UID: "6de126ef-158b-46a2-ae50-8e664270679d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:05.254110 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.254075    2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-workload-certs\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:05.254110 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.254100    2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-workload-socket\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:05.254409 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.254116    2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6de126ef-158b-46a2-ae50-8e664270679d-istiod-ca-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:05.254409 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.254130    2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-credential-socket\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:05.254409 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.254144    2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-istio-data\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:05.255484 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.255456    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de126ef-158b-46a2-ae50-8e664270679d-istio-token" (OuterVolumeSpecName: "istio-token") pod "6de126ef-158b-46a2-ae50-8e664270679d" (UID: "6de126ef-158b-46a2-ae50-8e664270679d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:44:05.255782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.255755    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6de126ef-158b-46a2-ae50-8e664270679d" (UID: "6de126ef-158b-46a2-ae50-8e664270679d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:05.255866 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.255844    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de126ef-158b-46a2-ae50-8e664270679d-kube-api-access-dr7zx" (OuterVolumeSpecName: "kube-api-access-dr7zx") pod "6de126ef-158b-46a2-ae50-8e664270679d" (UID: "6de126ef-158b-46a2-ae50-8e664270679d"). InnerVolumeSpecName "kube-api-access-dr7zx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:44:05.256094 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.256074    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6de126ef-158b-46a2-ae50-8e664270679d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6de126ef-158b-46a2-ae50-8e664270679d" (UID: "6de126ef-158b-46a2-ae50-8e664270679d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 13:44:05.334685 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.334653    2586 generic.go:356] "Generic (PLEG): container finished" podID="6de126ef-158b-46a2-ae50-8e664270679d" containerID="a9be7ab35103fdb18b34ed8d39b04c409a6b13d4ad174f4279831cbe4e9ec17c" exitCode=0
Sep 18 13:44:05.334859 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.334701    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"
Sep 18 13:44:05.334859 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.334733    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z" event={"ID":"6de126ef-158b-46a2-ae50-8e664270679d","Type":"ContainerDied","Data":"a9be7ab35103fdb18b34ed8d39b04c409a6b13d4ad174f4279831cbe4e9ec17c"}
Sep 18 13:44:05.334859 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.334782    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z" event={"ID":"6de126ef-158b-46a2-ae50-8e664270679d","Type":"ContainerDied","Data":"dbb9920cc04af7b58b7cbd27a0e46346b019612119601c7b43b5c2fc69818772"}
Sep 18 13:44:05.334859 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.334806    2586 scope.go:117] "RemoveContainer" containerID="a9be7ab35103fdb18b34ed8d39b04c409a6b13d4ad174f4279831cbe4e9ec17c"
Sep 18 13:44:05.349460 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.349443    2586 scope.go:117] "RemoveContainer" containerID="2197ee1f06e795c0a342c2780e122dbc0745c504da97cb5b4ee1a91880602952"
Sep 18 13:44:05.355404 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.355380    2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6de126ef-158b-46a2-ae50-8e664270679d-istio-envoy\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:05.355404 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.355407    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr7zx\" (UniqueName: \"kubernetes.io/projected/6de126ef-158b-46a2-ae50-8e664270679d-kube-api-access-dr7zx\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:05.355535 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.355416    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6de126ef-158b-46a2-ae50-8e664270679d-istio-token\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:05.355535 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.355425    2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6de126ef-158b-46a2-ae50-8e664270679d-istio-podinfo\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:05.357830 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.357818    2586 scope.go:117] "RemoveContainer" containerID="a9be7ab35103fdb18b34ed8d39b04c409a6b13d4ad174f4279831cbe4e9ec17c"
Sep 18 13:44:05.358158 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:44:05.358135    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9be7ab35103fdb18b34ed8d39b04c409a6b13d4ad174f4279831cbe4e9ec17c\": container with ID starting with a9be7ab35103fdb18b34ed8d39b04c409a6b13d4ad174f4279831cbe4e9ec17c not found: ID does not exist" containerID="a9be7ab35103fdb18b34ed8d39b04c409a6b13d4ad174f4279831cbe4e9ec17c"
Sep 18 13:44:05.358215 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.358166    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9be7ab35103fdb18b34ed8d39b04c409a6b13d4ad174f4279831cbe4e9ec17c"} err="failed to get container status \"a9be7ab35103fdb18b34ed8d39b04c409a6b13d4ad174f4279831cbe4e9ec17c\": rpc error: code = NotFound desc = could not find container \"a9be7ab35103fdb18b34ed8d39b04c409a6b13d4ad174f4279831cbe4e9ec17c\": container with ID starting with a9be7ab35103fdb18b34ed8d39b04c409a6b13d4ad174f4279831cbe4e9ec17c not found: ID does not exist"
Sep 18 13:44:05.358215 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.358184    2586 scope.go:117] "RemoveContainer" containerID="2197ee1f06e795c0a342c2780e122dbc0745c504da97cb5b4ee1a91880602952"
Sep 18 13:44:05.358448 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:44:05.358432    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2197ee1f06e795c0a342c2780e122dbc0745c504da97cb5b4ee1a91880602952\": container with ID starting with 2197ee1f06e795c0a342c2780e122dbc0745c504da97cb5b4ee1a91880602952 not found: ID does not exist" containerID="2197ee1f06e795c0a342c2780e122dbc0745c504da97cb5b4ee1a91880602952"
Sep 18 13:44:05.358502 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.358453    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2197ee1f06e795c0a342c2780e122dbc0745c504da97cb5b4ee1a91880602952"} err="failed to get container status \"2197ee1f06e795c0a342c2780e122dbc0745c504da97cb5b4ee1a91880602952\": rpc error: code = NotFound desc = could not find container \"2197ee1f06e795c0a342c2780e122dbc0745c504da97cb5b4ee1a91880602952\": container with ID starting with 2197ee1f06e795c0a342c2780e122dbc0745c504da97cb5b4ee1a91880602952 not found: ID does not exist"
Sep 18 13:44:05.367957 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.367933    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"]
Sep 18 13:44:05.384875 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:05.384846    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-57732b-openshift-default-759dbc7b86-kqp5z"]
Sep 18 13:44:06.850594 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:06.850555    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6de126ef-158b-46a2-ae50-8e664270679d" path="/var/lib/kubelet/pods/6de126ef-158b-46a2-ae50-8e664270679d/volumes"
Sep 18 13:44:08.843796 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:08.843731    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w" podUID="ce712460-0d49-4b14-ad89-67160b0c1320" containerName="main" containerID="cri-o://7fd8025741f740e374c50cd3c94d1f0ceb70908529badb6c583ce55daa870264" gracePeriod=45
Sep 18 13:44:09.090844 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.090821    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:44:09.091872 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.091853    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce712460-0d49-4b14-ad89-67160b0c1320-model-cache\") pod \"ce712460-0d49-4b14-ad89-67160b0c1320\" (UID: \"ce712460-0d49-4b14-ad89-67160b0c1320\") "
Sep 18 13:44:09.091944 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.091892    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpdlh\" (UniqueName: \"kubernetes.io/projected/ce712460-0d49-4b14-ad89-67160b0c1320-kube-api-access-xpdlh\") pod \"ce712460-0d49-4b14-ad89-67160b0c1320\" (UID: \"ce712460-0d49-4b14-ad89-67160b0c1320\") "
Sep 18 13:44:09.091992 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.091953    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce712460-0d49-4b14-ad89-67160b0c1320-tls-certs\") pod \"ce712460-0d49-4b14-ad89-67160b0c1320\" (UID: \"ce712460-0d49-4b14-ad89-67160b0c1320\") "
Sep 18 13:44:09.091992 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.091980    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce712460-0d49-4b14-ad89-67160b0c1320-tmp-dir\") pod \"ce712460-0d49-4b14-ad89-67160b0c1320\" (UID: \"ce712460-0d49-4b14-ad89-67160b0c1320\") "
Sep 18 13:44:09.092094 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.092040    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce712460-0d49-4b14-ad89-67160b0c1320-home\") pod \"ce712460-0d49-4b14-ad89-67160b0c1320\" (UID: \"ce712460-0d49-4b14-ad89-67160b0c1320\") "
Sep 18 13:44:09.092094 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.092071    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce712460-0d49-4b14-ad89-67160b0c1320-dshm\") pod \"ce712460-0d49-4b14-ad89-67160b0c1320\" (UID: \"ce712460-0d49-4b14-ad89-67160b0c1320\") "
Sep 18 13:44:09.092094 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.092071    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce712460-0d49-4b14-ad89-67160b0c1320-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce712460-0d49-4b14-ad89-67160b0c1320" (UID: "ce712460-0d49-4b14-ad89-67160b0c1320"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:09.092319 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.092290    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce712460-0d49-4b14-ad89-67160b0c1320-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ce712460-0d49-4b14-ad89-67160b0c1320" (UID: "ce712460-0d49-4b14-ad89-67160b0c1320"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:09.092319 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.092309    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce712460-0d49-4b14-ad89-67160b0c1320-home" (OuterVolumeSpecName: "home") pod "ce712460-0d49-4b14-ad89-67160b0c1320" (UID: "ce712460-0d49-4b14-ad89-67160b0c1320"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:09.092481 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.092379    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce712460-0d49-4b14-ad89-67160b0c1320-model-cache\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:09.092481 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.092412    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce712460-0d49-4b14-ad89-67160b0c1320-tmp-dir\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:09.092481 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.092421    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce712460-0d49-4b14-ad89-67160b0c1320-home\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:09.094249 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.094195    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce712460-0d49-4b14-ad89-67160b0c1320-kube-api-access-xpdlh" (OuterVolumeSpecName: "kube-api-access-xpdlh") pod "ce712460-0d49-4b14-ad89-67160b0c1320" (UID: "ce712460-0d49-4b14-ad89-67160b0c1320"). InnerVolumeSpecName "kube-api-access-xpdlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:44:09.094249 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.094235    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce712460-0d49-4b14-ad89-67160b0c1320-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce712460-0d49-4b14-ad89-67160b0c1320" (UID: "ce712460-0d49-4b14-ad89-67160b0c1320"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:44:09.094389 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.094240    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce712460-0d49-4b14-ad89-67160b0c1320-dshm" (OuterVolumeSpecName: "dshm") pod "ce712460-0d49-4b14-ad89-67160b0c1320" (UID: "ce712460-0d49-4b14-ad89-67160b0c1320"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:44:09.193100 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.193065    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpdlh\" (UniqueName: \"kubernetes.io/projected/ce712460-0d49-4b14-ad89-67160b0c1320-kube-api-access-xpdlh\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:09.193100 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.193097    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce712460-0d49-4b14-ad89-67160b0c1320-tls-certs\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:09.193100 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.193108    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce712460-0d49-4b14-ad89-67160b0c1320-dshm\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:09.360045 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.359957    2586 generic.go:356] "Generic (PLEG): container finished" podID="ce712460-0d49-4b14-ad89-67160b0c1320" containerID="7fd8025741f740e374c50cd3c94d1f0ceb70908529badb6c583ce55daa870264" exitCode=0
Sep 18 13:44:09.360045 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.360019    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"
Sep 18 13:44:09.360273 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.360045    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w" event={"ID":"ce712460-0d49-4b14-ad89-67160b0c1320","Type":"ContainerDied","Data":"7fd8025741f740e374c50cd3c94d1f0ceb70908529badb6c583ce55daa870264"}
Sep 18 13:44:09.360273 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.360104    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w" event={"ID":"ce712460-0d49-4b14-ad89-67160b0c1320","Type":"ContainerDied","Data":"37c2ee38fdd2e6d36d63b6a1688ba74134c5deb1dc6c02eec2abecd95733f57a"}
Sep 18 13:44:09.360273 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.360128    2586 scope.go:117] "RemoveContainer" containerID="7fd8025741f740e374c50cd3c94d1f0ceb70908529badb6c583ce55daa870264"
Sep 18 13:44:09.370183 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.370168    2586 scope.go:117] "RemoveContainer" containerID="7fd8025741f740e374c50cd3c94d1f0ceb70908529badb6c583ce55daa870264"
Sep 18 13:44:09.370442 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:44:09.370419    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fd8025741f740e374c50cd3c94d1f0ceb70908529badb6c583ce55daa870264\": container with ID starting with 7fd8025741f740e374c50cd3c94d1f0ceb70908529badb6c583ce55daa870264 not found: ID does not exist" containerID="7fd8025741f740e374c50cd3c94d1f0ceb70908529badb6c583ce55daa870264"
Sep 18 13:44:09.370512 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.370449    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fd8025741f740e374c50cd3c94d1f0ceb70908529badb6c583ce55daa870264"} err="failed to get container status \"7fd8025741f740e374c50cd3c94d1f0ceb70908529badb6c583ce55daa870264\": rpc error: code = NotFound desc = could not find container \"7fd8025741f740e374c50cd3c94d1f0ceb70908529badb6c583ce55daa870264\": container with ID starting with 7fd8025741f740e374c50cd3c94d1f0ceb70908529badb6c583ce55daa870264 not found: ID does not exist"
Sep 18 13:44:09.401644 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.401611    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"]
Sep 18 13:44:09.408476 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:09.408455    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-951469/premium-w0-951469-kserve-b757b4d6b-9j72w"]
Sep 18 13:44:10.075967 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.075934    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-951469-29828970-tpw7s"]
Sep 18 13:44:10.086155 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.086122    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-951469-29828970-tpw7s"]
Sep 18 13:44:10.129336 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.129304    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng"]
Sep 18 13:44:10.129571 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.129544    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng" podUID="b8d4efe1-5e52-4f7d-8d17-22715cd25947" containerName="payload-processing" containerID="cri-o://69c32c1e132a4fd53d62ca5a5b5d3a05fd468bc190b0bb0eb251e710cfc6b420" gracePeriod=30
Sep 18 13:44:10.142040 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.142016    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw"]
Sep 18 13:44:10.142274 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.142243    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw" podUID="3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3" containerName="payload-pre-processing" containerID="cri-o://0b40f7e1b819c757b49cb357ab6334240455213259de4a7b560eb886aa5dadbd" gracePeriod=30
Sep 18 13:44:10.369767 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.369738    2586 generic.go:356] "Generic (PLEG): container finished" podID="b8d4efe1-5e52-4f7d-8d17-22715cd25947" containerID="69c32c1e132a4fd53d62ca5a5b5d3a05fd468bc190b0bb0eb251e710cfc6b420" exitCode=0
Sep 18 13:44:10.369952 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.369818    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng" event={"ID":"b8d4efe1-5e52-4f7d-8d17-22715cd25947","Type":"ContainerDied","Data":"69c32c1e132a4fd53d62ca5a5b5d3a05fd468bc190b0bb0eb251e710cfc6b420"}
Sep 18 13:44:10.371804 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.371783    2586 generic.go:356] "Generic (PLEG): container finished" podID="3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3" containerID="0b40f7e1b819c757b49cb357ab6334240455213259de4a7b560eb886aa5dadbd" exitCode=0
Sep 18 13:44:10.371943 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.371866    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw" event={"ID":"3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3","Type":"ContainerDied","Data":"0b40f7e1b819c757b49cb357ab6334240455213259de4a7b560eb886aa5dadbd"}
Sep 18 13:44:10.436416 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.436394    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw"
Sep 18 13:44:10.439925 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.439889    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng"
Sep 18 13:44:10.476961 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.476931    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl"]
Sep 18 13:44:10.477158 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.477126    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl" podUID="b38d31ed-2a8c-409c-9fc8-adcf2ba72730" containerName="payload-processing" containerID="cri-o://1cdcbcf70302ead0828401706bebece7537c27565c6106bd6b5609130dadf85b" gracePeriod=30
Sep 18 13:44:10.501733 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.501710    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl"]
Sep 18 13:44:10.501825 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.501774    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96lv8\" (UniqueName: \"kubernetes.io/projected/3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3-kube-api-access-96lv8\") pod \"3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3\" (UID: \"3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3\") "
Sep 18 13:44:10.501890 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.501829    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qz7sk\" (UniqueName: \"kubernetes.io/projected/b8d4efe1-5e52-4f7d-8d17-22715cd25947-kube-api-access-qz7sk\") pod \"b8d4efe1-5e52-4f7d-8d17-22715cd25947\" (UID: \"b8d4efe1-5e52-4f7d-8d17-22715cd25947\") "
Sep 18 13:44:10.501890 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.501876    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3-plugins-config-volume\") pod \"3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3\" (UID: \"3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3\") "
Sep 18 13:44:10.502056 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.501879    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl" podUID="96bc2e44-afe8-42f5-a5ab-0c8c95752ed3" containerName="payload-pre-processing" containerID="cri-o://064cbf4250713f3e3f94d1ed1963f281f45c64c4daec6f9c2416fec68c5a5d76" gracePeriod=30
Sep 18 13:44:10.502056 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.501941    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d4efe1-5e52-4f7d-8d17-22715cd25947-plugins-config-volume\") pod \"b8d4efe1-5e52-4f7d-8d17-22715cd25947\" (UID: \"b8d4efe1-5e52-4f7d-8d17-22715cd25947\") "
Sep 18 13:44:10.502363 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.502325    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3" (UID: "3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:44:10.502455 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.502391    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d4efe1-5e52-4f7d-8d17-22715cd25947-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b8d4efe1-5e52-4f7d-8d17-22715cd25947" (UID: "b8d4efe1-5e52-4f7d-8d17-22715cd25947"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:44:10.506988 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.505736    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d4efe1-5e52-4f7d-8d17-22715cd25947-kube-api-access-qz7sk" (OuterVolumeSpecName: "kube-api-access-qz7sk") pod "b8d4efe1-5e52-4f7d-8d17-22715cd25947" (UID: "b8d4efe1-5e52-4f7d-8d17-22715cd25947"). InnerVolumeSpecName "kube-api-access-qz7sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:44:10.506988 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.505977    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3-kube-api-access-96lv8" (OuterVolumeSpecName: "kube-api-access-96lv8") pod "3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3" (UID: "3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3"). InnerVolumeSpecName "kube-api-access-96lv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:44:10.602859 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.602797    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3-plugins-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:10.602859 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.602822    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8d4efe1-5e52-4f7d-8d17-22715cd25947-plugins-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:10.602859 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.602833    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96lv8\" (UniqueName: \"kubernetes.io/projected/3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3-kube-api-access-96lv8\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:10.602859 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.602841    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz7sk\" (UniqueName: \"kubernetes.io/projected/b8d4efe1-5e52-4f7d-8d17-22715cd25947-kube-api-access-qz7sk\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:10.765697 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.765674    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl"
Sep 18 13:44:10.772357 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.772334    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl"
Sep 18 13:44:10.804011 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.803981    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96bc2e44-afe8-42f5-a5ab-0c8c95752ed3-plugins-config-volume\") pod \"96bc2e44-afe8-42f5-a5ab-0c8c95752ed3\" (UID: \"96bc2e44-afe8-42f5-a5ab-0c8c95752ed3\") "
Sep 18 13:44:10.804143 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.804029    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38d31ed-2a8c-409c-9fc8-adcf2ba72730-plugins-config-volume\") pod \"b38d31ed-2a8c-409c-9fc8-adcf2ba72730\" (UID: \"b38d31ed-2a8c-409c-9fc8-adcf2ba72730\") "
Sep 18 13:44:10.804143 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.804078    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46mfk\" (UniqueName: \"kubernetes.io/projected/b38d31ed-2a8c-409c-9fc8-adcf2ba72730-kube-api-access-46mfk\") pod \"b38d31ed-2a8c-409c-9fc8-adcf2ba72730\" (UID: \"b38d31ed-2a8c-409c-9fc8-adcf2ba72730\") "
Sep 18 13:44:10.804143 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.804140    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8d2w\" (UniqueName: \"kubernetes.io/projected/96bc2e44-afe8-42f5-a5ab-0c8c95752ed3-kube-api-access-t8d2w\") pod \"96bc2e44-afe8-42f5-a5ab-0c8c95752ed3\" (UID: \"96bc2e44-afe8-42f5-a5ab-0c8c95752ed3\") "
Sep 18 13:44:10.804707 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.804412    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96bc2e44-afe8-42f5-a5ab-0c8c95752ed3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "96bc2e44-afe8-42f5-a5ab-0c8c95752ed3" (UID: "96bc2e44-afe8-42f5-a5ab-0c8c95752ed3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:44:10.805080 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.805047    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b38d31ed-2a8c-409c-9fc8-adcf2ba72730-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b38d31ed-2a8c-409c-9fc8-adcf2ba72730" (UID: "b38d31ed-2a8c-409c-9fc8-adcf2ba72730"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:44:10.807562 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.807521    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b38d31ed-2a8c-409c-9fc8-adcf2ba72730-kube-api-access-46mfk" (OuterVolumeSpecName: "kube-api-access-46mfk") pod "b38d31ed-2a8c-409c-9fc8-adcf2ba72730" (UID: "b38d31ed-2a8c-409c-9fc8-adcf2ba72730"). InnerVolumeSpecName "kube-api-access-46mfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:44:10.807652 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.807614    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96bc2e44-afe8-42f5-a5ab-0c8c95752ed3-kube-api-access-t8d2w" (OuterVolumeSpecName: "kube-api-access-t8d2w") pod "96bc2e44-afe8-42f5-a5ab-0c8c95752ed3" (UID: "96bc2e44-afe8-42f5-a5ab-0c8c95752ed3"). InnerVolumeSpecName "kube-api-access-t8d2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:44:10.851413 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.851377    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8b28228-52f3-4edd-b1af-15438f5ea33b" path="/var/lib/kubelet/pods/b8b28228-52f3-4edd-b1af-15438f5ea33b/volumes"
Sep 18 13:44:10.851795 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.851775    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce712460-0d49-4b14-ad89-67160b0c1320" path="/var/lib/kubelet/pods/ce712460-0d49-4b14-ad89-67160b0c1320/volumes"
Sep 18 13:44:10.905877 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.905854    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8d2w\" (UniqueName: \"kubernetes.io/projected/96bc2e44-afe8-42f5-a5ab-0c8c95752ed3-kube-api-access-t8d2w\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:10.906060 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.906042    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96bc2e44-afe8-42f5-a5ab-0c8c95752ed3-plugins-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:10.906137 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.906108    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b38d31ed-2a8c-409c-9fc8-adcf2ba72730-plugins-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:10.906137 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:10.906125    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46mfk\" (UniqueName: \"kubernetes.io/projected/b38d31ed-2a8c-409c-9fc8-adcf2ba72730-kube-api-access-46mfk\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:11.097061 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.097034    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b2d5da-bf6pd"]
Sep 18 13:44:11.106323 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.106295    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b2d5da-bf6pd"]
Sep 18 13:44:11.378484 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.378446    2586 generic.go:356] "Generic (PLEG): container finished" podID="b38d31ed-2a8c-409c-9fc8-adcf2ba72730" containerID="1cdcbcf70302ead0828401706bebece7537c27565c6106bd6b5609130dadf85b" exitCode=0
Sep 18 13:44:11.378642 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.378485    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl" event={"ID":"b38d31ed-2a8c-409c-9fc8-adcf2ba72730","Type":"ContainerDied","Data":"1cdcbcf70302ead0828401706bebece7537c27565c6106bd6b5609130dadf85b"}
Sep 18 13:44:11.378642 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.378509    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl"
Sep 18 13:44:11.378642 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.378563    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl" event={"ID":"b38d31ed-2a8c-409c-9fc8-adcf2ba72730","Type":"ContainerDied","Data":"94494c9ea4943ec27ff74a286f488b550ed2c5942fa0c7a3ac33275e50a5cc59"}
Sep 18 13:44:11.378642 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.378595    2586 scope.go:117] "RemoveContainer" containerID="1cdcbcf70302ead0828401706bebece7537c27565c6106bd6b5609130dadf85b"
Sep 18 13:44:11.379988 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.379856    2586 generic.go:356] "Generic (PLEG): container finished" podID="96bc2e44-afe8-42f5-a5ab-0c8c95752ed3" containerID="064cbf4250713f3e3f94d1ed1963f281f45c64c4daec6f9c2416fec68c5a5d76" exitCode=0
Sep 18 13:44:11.379988 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.379898    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl"
Sep 18 13:44:11.379988 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.379944    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl" event={"ID":"96bc2e44-afe8-42f5-a5ab-0c8c95752ed3","Type":"ContainerDied","Data":"064cbf4250713f3e3f94d1ed1963f281f45c64c4daec6f9c2416fec68c5a5d76"}
Sep 18 13:44:11.380159 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.379992    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl" event={"ID":"96bc2e44-afe8-42f5-a5ab-0c8c95752ed3","Type":"ContainerDied","Data":"731ded0e8a360a95cdadcf4ca3e4321239bf5319d039d7fd202aef9e8eb0463d"}
Sep 18 13:44:11.381122 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.381001    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw" podUID="3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.27:9004: i/o timeout"
Sep 18 13:44:11.381442 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.381410    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng" podUID="b8d4efe1-5e52-4f7d-8d17-22715cd25947" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.28:9004: i/o timeout"
Sep 18 13:44:11.381770 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.381730    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng" event={"ID":"b8d4efe1-5e52-4f7d-8d17-22715cd25947","Type":"ContainerDied","Data":"90bbc90b6165c489562cbf2c97d96cbd09ea3f214d8c32ecc9e3f0ed41405fc6"}
Sep 18 13:44:11.381770 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.381761    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng"
Sep 18 13:44:11.383152 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.383128    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw"
Sep 18 13:44:11.383152 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.383147    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw" event={"ID":"3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3","Type":"ContainerDied","Data":"f52e07b1cd3f2b8e7ba774769d992eee99903debf252a728fa8419fa2648435a"}
Sep 18 13:44:11.388945 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.388928    2586 scope.go:117] "RemoveContainer" containerID="1cdcbcf70302ead0828401706bebece7537c27565c6106bd6b5609130dadf85b"
Sep 18 13:44:11.389212 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:44:11.389181    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cdcbcf70302ead0828401706bebece7537c27565c6106bd6b5609130dadf85b\": container with ID starting with 1cdcbcf70302ead0828401706bebece7537c27565c6106bd6b5609130dadf85b not found: ID does not exist" containerID="1cdcbcf70302ead0828401706bebece7537c27565c6106bd6b5609130dadf85b"
Sep 18 13:44:11.389308 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.389213    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cdcbcf70302ead0828401706bebece7537c27565c6106bd6b5609130dadf85b"} err="failed to get container status \"1cdcbcf70302ead0828401706bebece7537c27565c6106bd6b5609130dadf85b\": rpc error: code = NotFound desc = could not find container \"1cdcbcf70302ead0828401706bebece7537c27565c6106bd6b5609130dadf85b\": container with ID starting with 1cdcbcf70302ead0828401706bebece7537c27565c6106bd6b5609130dadf85b not found: ID does not exist"
Sep 18 13:44:11.389308 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.389235    2586 scope.go:117] "RemoveContainer" containerID="064cbf4250713f3e3f94d1ed1963f281f45c64c4daec6f9c2416fec68c5a5d76"
Sep 18 13:44:11.396811 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.396797    2586 scope.go:117] "RemoveContainer" containerID="064cbf4250713f3e3f94d1ed1963f281f45c64c4daec6f9c2416fec68c5a5d76"
Sep 18 13:44:11.397086 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:44:11.397066    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"064cbf4250713f3e3f94d1ed1963f281f45c64c4daec6f9c2416fec68c5a5d76\": container with ID starting with 064cbf4250713f3e3f94d1ed1963f281f45c64c4daec6f9c2416fec68c5a5d76 not found: ID does not exist" containerID="064cbf4250713f3e3f94d1ed1963f281f45c64c4daec6f9c2416fec68c5a5d76"
Sep 18 13:44:11.397150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.397093    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"064cbf4250713f3e3f94d1ed1963f281f45c64c4daec6f9c2416fec68c5a5d76"} err="failed to get container status \"064cbf4250713f3e3f94d1ed1963f281f45c64c4daec6f9c2416fec68c5a5d76\": rpc error: code = NotFound desc = could not find container \"064cbf4250713f3e3f94d1ed1963f281f45c64c4daec6f9c2416fec68c5a5d76\": container with ID starting with 064cbf4250713f3e3f94d1ed1963f281f45c64c4daec6f9c2416fec68c5a5d76 not found: ID does not exist"
Sep 18 13:44:11.397150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.397108    2586 scope.go:117] "RemoveContainer" containerID="69c32c1e132a4fd53d62ca5a5b5d3a05fd468bc190b0bb0eb251e710cfc6b420"
Sep 18 13:44:11.405560 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.405537    2586 scope.go:117] "RemoveContainer" containerID="0b40f7e1b819c757b49cb357ab6334240455213259de4a7b560eb886aa5dadbd"
Sep 18 13:44:11.416069 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.416048    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw"]
Sep 18 13:44:11.423020 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.423001    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-951469-5cbfc7c6db-7rrqw"]
Sep 18 13:44:11.436293 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.436268    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl"]
Sep 18 13:44:11.441059 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.441039    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b2d5da-8d9d76446-5swzl"]
Sep 18 13:44:11.463680 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.463662    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl"]
Sep 18 13:44:11.480114 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.480094    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b2d5da-58c5db6457-g4rzl"]
Sep 18 13:44:11.501823 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.501800    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng"]
Sep 18 13:44:11.506070 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:11.506032    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-951469-754859f7dc-vvjng"]
Sep 18 13:44:12.849484 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:12.849445    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3" path="/var/lib/kubelet/pods/3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3/volumes"
Sep 18 13:44:12.849944 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:12.849769    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cdc05c5-1d49-40c2-9e36-edf6ef89a945" path="/var/lib/kubelet/pods/8cdc05c5-1d49-40c2-9e36-edf6ef89a945/volumes"
Sep 18 13:44:12.850139 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:12.850124    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96bc2e44-afe8-42f5-a5ab-0c8c95752ed3" path="/var/lib/kubelet/pods/96bc2e44-afe8-42f5-a5ab-0c8c95752ed3/volumes"
Sep 18 13:44:12.850680 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:12.850653    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b38d31ed-2a8c-409c-9fc8-adcf2ba72730" path="/var/lib/kubelet/pods/b38d31ed-2a8c-409c-9fc8-adcf2ba72730/volumes"
Sep 18 13:44:12.851261 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:12.851244    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8d4efe1-5e52-4f7d-8d17-22715cd25947" path="/var/lib/kubelet/pods/b8d4efe1-5e52-4f7d-8d17-22715cd25947/volumes"
Sep 18 13:44:20.566711 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.566675    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92b54f-h24sm"]
Sep 18 13:44:20.567124 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.567050    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce712460-0d49-4b14-ad89-67160b0c1320" containerName="main"
Sep 18 13:44:20.567124 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.567061    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce712460-0d49-4b14-ad89-67160b0c1320" containerName="main"
Sep 18 13:44:20.567124 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.567070    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b38d31ed-2a8c-409c-9fc8-adcf2ba72730" containerName="payload-processing"
Sep 18 13:44:20.567124 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.567076    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b38d31ed-2a8c-409c-9fc8-adcf2ba72730" containerName="payload-processing"
Sep 18 13:44:20.567124 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.567085    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3" containerName="payload-pre-processing"
Sep 18 13:44:20.567124 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.567090    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3" containerName="payload-pre-processing"
Sep 18 13:44:20.567124 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.567097    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cdc05c5-1d49-40c2-9e36-edf6ef89a945" containerName="revoke-keys"
Sep 18 13:44:20.567124 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.567102    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cdc05c5-1d49-40c2-9e36-edf6ef89a945" containerName="revoke-keys"
Sep 18 13:44:20.567124 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.567115    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6de126ef-158b-46a2-ae50-8e664270679d" containerName="istio-proxy"
Sep 18 13:44:20.567124 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.567120    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6de126ef-158b-46a2-ae50-8e664270679d" containerName="istio-proxy"
Sep 18 13:44:20.567124 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.567130    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8d4efe1-5e52-4f7d-8d17-22715cd25947" containerName="payload-processing"
Sep 18 13:44:20.567504 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.567135    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8d4efe1-5e52-4f7d-8d17-22715cd25947" containerName="payload-processing"
Sep 18 13:44:20.567504 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.567147    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96bc2e44-afe8-42f5-a5ab-0c8c95752ed3" containerName="payload-pre-processing"
Sep 18 13:44:20.567504 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.567152    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="96bc2e44-afe8-42f5-a5ab-0c8c95752ed3" containerName="payload-pre-processing"
Sep 18 13:44:20.567504 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.567205    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="96bc2e44-afe8-42f5-a5ab-0c8c95752ed3" containerName="payload-pre-processing"
Sep 18 13:44:20.567504 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.567213    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b38d31ed-2a8c-409c-9fc8-adcf2ba72730" containerName="payload-processing"
Sep 18 13:44:20.567504 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.567218    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce712460-0d49-4b14-ad89-67160b0c1320" containerName="main"
Sep 18 13:44:20.567504 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.567226    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cdc05c5-1d49-40c2-9e36-edf6ef89a945" containerName="revoke-keys"
Sep 18 13:44:20.567504 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.567232    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6de126ef-158b-46a2-ae50-8e664270679d" containerName="istio-proxy"
Sep 18 13:44:20.567504 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.567238    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e9a1ca8-e65e-4d6b-b4fc-9ee6c4ec4ad3" containerName="payload-pre-processing"
Sep 18 13:44:20.567504 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.567245    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8d4efe1-5e52-4f7d-8d17-22715cd25947" containerName="payload-processing"
Sep 18 13:44:20.571804 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.571787    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92b54f-h24sm"
Sep 18 13:44:20.573823 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.573803    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:44:20.574216 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.574202    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nh2cd\""
Sep 18 13:44:20.578935 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.578852    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92b54f-h24sm"]
Sep 18 13:44:20.598427 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.598401    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcddd9ad-7afb-48b3-8aa2-61ef12ea0c97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-92b54f-h24sm\" (UID: \"dcddd9ad-7afb-48b3-8aa2-61ef12ea0c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92b54f-h24sm"
Sep 18 13:44:20.699523 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.699492    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcddd9ad-7afb-48b3-8aa2-61ef12ea0c97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-92b54f-h24sm\" (UID: \"dcddd9ad-7afb-48b3-8aa2-61ef12ea0c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92b54f-h24sm"
Sep 18 13:44:20.700159 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.700137    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcddd9ad-7afb-48b3-8aa2-61ef12ea0c97-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-92b54f-h24sm\" (UID: \"dcddd9ad-7afb-48b3-8aa2-61ef12ea0c97\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92b54f-h24sm"
Sep 18 13:44:20.884051 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.883972    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92b54f-h24sm"
Sep 18 13:44:21.000025 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:20.999994    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:44:21.173248 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:21.173181    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92b54f-h24sm"]
Sep 18 13:44:21.175539 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:44:21.175511    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcddd9ad_7afb_48b3_8aa2_61ef12ea0c97.slice/crio-e5132eb75f88659e486411a8947d05b16d8ef77daa98a4595de313a983cec5d9 WatchSource:0}: Error finding container e5132eb75f88659e486411a8947d05b16d8ef77daa98a4595de313a983cec5d9: Status 404 returned error can't find the container with id e5132eb75f88659e486411a8947d05b16d8ef77daa98a4595de313a983cec5d9
Sep 18 13:44:21.439825 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:21.439758    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92b54f-h24sm" event={"ID":"dcddd9ad-7afb-48b3-8aa2-61ef12ea0c97","Type":"ContainerStarted","Data":"e5132eb75f88659e486411a8947d05b16d8ef77daa98a4595de313a983cec5d9"}
Sep 18 13:44:22.447037 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:22.447002    2586 generic.go:356] "Generic (PLEG): container finished" podID="dcddd9ad-7afb-48b3-8aa2-61ef12ea0c97" containerID="baacfe80e4c697a8ee7ad30866edd86512476d68e0b428ce237c9a95ef4b804c" exitCode=0
Sep 18 13:44:22.447396 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:22.447085    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92b54f-h24sm" event={"ID":"dcddd9ad-7afb-48b3-8aa2-61ef12ea0c97","Type":"ContainerDied","Data":"baacfe80e4c697a8ee7ad30866edd86512476d68e0b428ce237c9a95ef4b804c"}
Sep 18 13:44:23.586074 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:23.586049    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92b54f-h24sm"
Sep 18 13:44:23.627176 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:23.627143    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcddd9ad-7afb-48b3-8aa2-61ef12ea0c97-maas-api-service-ca\") pod \"dcddd9ad-7afb-48b3-8aa2-61ef12ea0c97\" (UID: \"dcddd9ad-7afb-48b3-8aa2-61ef12ea0c97\") "
Sep 18 13:44:23.627482 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:23.627460    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcddd9ad-7afb-48b3-8aa2-61ef12ea0c97-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dcddd9ad-7afb-48b3-8aa2-61ef12ea0c97" (UID: "dcddd9ad-7afb-48b3-8aa2-61ef12ea0c97"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:44:23.728861 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:23.728782    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcddd9ad-7afb-48b3-8aa2-61ef12ea0c97-maas-api-service-ca\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:24.459829 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:24.459794    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92b54f-h24sm"
Sep 18 13:44:24.460018 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:24.459837    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92b54f-h24sm" event={"ID":"dcddd9ad-7afb-48b3-8aa2-61ef12ea0c97","Type":"ContainerDied","Data":"e5132eb75f88659e486411a8947d05b16d8ef77daa98a4595de313a983cec5d9"}
Sep 18 13:44:24.460018 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:24.459882    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5132eb75f88659e486411a8947d05b16d8ef77daa98a4595de313a983cec5d9"
Sep 18 13:44:34.086938 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.086888    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv"]
Sep 18 13:44:34.087384 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.087141    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv" podUID="2b873220-afcc-4185-ae47-d8b853b07971" containerName="payload-processing" containerID="cri-o://5d2510e9511101f3c31e644d11e47ad8ac0210ca3b45db430d3d5ebaeb8b0ab1" gracePeriod=30
Sep 18 13:44:34.107424 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.107399    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9"]
Sep 18 13:44:34.107636 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.107592    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9" podUID="64e08c40-2d69-494e-a0e3-aad00fb5fa6a" containerName="payload-pre-processing" containerID="cri-o://6f106cc68ea5aca849fd98e90f5c0deb2382cfce8abb0e4a42d572eecec7771d" gracePeriod=30
Sep 18 13:44:34.372499 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.372471    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv"
Sep 18 13:44:34.375405 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.375384    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9"
Sep 18 13:44:34.431643 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.431610    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b873220-afcc-4185-ae47-d8b853b07971-plugins-config-volume\") pod \"2b873220-afcc-4185-ae47-d8b853b07971\" (UID: \"2b873220-afcc-4185-ae47-d8b853b07971\") "
Sep 18 13:44:34.431784 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.431674    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvbzz\" (UniqueName: \"kubernetes.io/projected/64e08c40-2d69-494e-a0e3-aad00fb5fa6a-kube-api-access-gvbzz\") pod \"64e08c40-2d69-494e-a0e3-aad00fb5fa6a\" (UID: \"64e08c40-2d69-494e-a0e3-aad00fb5fa6a\") "
Sep 18 13:44:34.431784 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.431723    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbc42\" (UniqueName: \"kubernetes.io/projected/2b873220-afcc-4185-ae47-d8b853b07971-kube-api-access-dbc42\") pod \"2b873220-afcc-4185-ae47-d8b853b07971\" (UID: \"2b873220-afcc-4185-ae47-d8b853b07971\") "
Sep 18 13:44:34.431784 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.431777    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e08c40-2d69-494e-a0e3-aad00fb5fa6a-plugins-config-volume\") pod \"64e08c40-2d69-494e-a0e3-aad00fb5fa6a\" (UID: \"64e08c40-2d69-494e-a0e3-aad00fb5fa6a\") "
Sep 18 13:44:34.431972 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.431954    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b873220-afcc-4185-ae47-d8b853b07971-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2b873220-afcc-4185-ae47-d8b853b07971" (UID: "2b873220-afcc-4185-ae47-d8b853b07971"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:44:34.432218 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.432195    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b873220-afcc-4185-ae47-d8b853b07971-plugins-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:34.432472 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.432197    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e08c40-2d69-494e-a0e3-aad00fb5fa6a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "64e08c40-2d69-494e-a0e3-aad00fb5fa6a" (UID: "64e08c40-2d69-494e-a0e3-aad00fb5fa6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:44:34.433774 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.433755    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e08c40-2d69-494e-a0e3-aad00fb5fa6a-kube-api-access-gvbzz" (OuterVolumeSpecName: "kube-api-access-gvbzz") pod "64e08c40-2d69-494e-a0e3-aad00fb5fa6a" (UID: "64e08c40-2d69-494e-a0e3-aad00fb5fa6a"). InnerVolumeSpecName "kube-api-access-gvbzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:44:34.433873 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.433825    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b873220-afcc-4185-ae47-d8b853b07971-kube-api-access-dbc42" (OuterVolumeSpecName: "kube-api-access-dbc42") pod "2b873220-afcc-4185-ae47-d8b853b07971" (UID: "2b873220-afcc-4185-ae47-d8b853b07971"). InnerVolumeSpecName "kube-api-access-dbc42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:44:34.513534 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.513494    2586 generic.go:356] "Generic (PLEG): container finished" podID="64e08c40-2d69-494e-a0e3-aad00fb5fa6a" containerID="6f106cc68ea5aca849fd98e90f5c0deb2382cfce8abb0e4a42d572eecec7771d" exitCode=0
Sep 18 13:44:34.513534 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.513532    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9"
Sep 18 13:44:34.513779 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.513571    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9" event={"ID":"64e08c40-2d69-494e-a0e3-aad00fb5fa6a","Type":"ContainerDied","Data":"6f106cc68ea5aca849fd98e90f5c0deb2382cfce8abb0e4a42d572eecec7771d"}
Sep 18 13:44:34.513920 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.513624    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9" event={"ID":"64e08c40-2d69-494e-a0e3-aad00fb5fa6a","Type":"ContainerDied","Data":"4a90a4fb7f5ccb98db9833e103b314d4327e73ad737a5905ed8e2021f8eb9c16"}
Sep 18 13:44:34.514016 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.513642    2586 scope.go:117] "RemoveContainer" containerID="6f106cc68ea5aca849fd98e90f5c0deb2382cfce8abb0e4a42d572eecec7771d"
Sep 18 13:44:34.525111 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.525057    2586 generic.go:356] "Generic (PLEG): container finished" podID="2b873220-afcc-4185-ae47-d8b853b07971" containerID="5d2510e9511101f3c31e644d11e47ad8ac0210ca3b45db430d3d5ebaeb8b0ab1" exitCode=0
Sep 18 13:44:34.525190 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.525110    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv" event={"ID":"2b873220-afcc-4185-ae47-d8b853b07971","Type":"ContainerDied","Data":"5d2510e9511101f3c31e644d11e47ad8ac0210ca3b45db430d3d5ebaeb8b0ab1"}
Sep 18 13:44:34.525190 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.525146    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv"
Sep 18 13:44:34.525309 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.525223    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv" event={"ID":"2b873220-afcc-4185-ae47-d8b853b07971","Type":"ContainerDied","Data":"494c76dcd7b5ec9e285e11b10230523624327fa6413878fbd2060f74a48af44f"}
Sep 18 13:44:34.533270 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.533246    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbc42\" (UniqueName: \"kubernetes.io/projected/2b873220-afcc-4185-ae47-d8b853b07971-kube-api-access-dbc42\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:34.533270 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.533270    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e08c40-2d69-494e-a0e3-aad00fb5fa6a-plugins-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:34.533420 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.533285    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvbzz\" (UniqueName: \"kubernetes.io/projected/64e08c40-2d69-494e-a0e3-aad00fb5fa6a-kube-api-access-gvbzz\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:44:34.534203 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.534187    2586 scope.go:117] "RemoveContainer" containerID="6f106cc68ea5aca849fd98e90f5c0deb2382cfce8abb0e4a42d572eecec7771d"
Sep 18 13:44:34.534481 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:44:34.534460    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f106cc68ea5aca849fd98e90f5c0deb2382cfce8abb0e4a42d572eecec7771d\": container with ID starting with 6f106cc68ea5aca849fd98e90f5c0deb2382cfce8abb0e4a42d572eecec7771d not found: ID does not exist" containerID="6f106cc68ea5aca849fd98e90f5c0deb2382cfce8abb0e4a42d572eecec7771d"
Sep 18 13:44:34.534544 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.534487    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f106cc68ea5aca849fd98e90f5c0deb2382cfce8abb0e4a42d572eecec7771d"} err="failed to get container status \"6f106cc68ea5aca849fd98e90f5c0deb2382cfce8abb0e4a42d572eecec7771d\": rpc error: code = NotFound desc = could not find container \"6f106cc68ea5aca849fd98e90f5c0deb2382cfce8abb0e4a42d572eecec7771d\": container with ID starting with 6f106cc68ea5aca849fd98e90f5c0deb2382cfce8abb0e4a42d572eecec7771d not found: ID does not exist"
Sep 18 13:44:34.534544 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.534504    2586 scope.go:117] "RemoveContainer" containerID="5d2510e9511101f3c31e644d11e47ad8ac0210ca3b45db430d3d5ebaeb8b0ab1"
Sep 18 13:44:34.542517 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.542501    2586 scope.go:117] "RemoveContainer" containerID="5d2510e9511101f3c31e644d11e47ad8ac0210ca3b45db430d3d5ebaeb8b0ab1"
Sep 18 13:44:34.542809 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:44:34.542790    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d2510e9511101f3c31e644d11e47ad8ac0210ca3b45db430d3d5ebaeb8b0ab1\": container with ID starting with 5d2510e9511101f3c31e644d11e47ad8ac0210ca3b45db430d3d5ebaeb8b0ab1 not found: ID does not exist" containerID="5d2510e9511101f3c31e644d11e47ad8ac0210ca3b45db430d3d5ebaeb8b0ab1"
Sep 18 13:44:34.542914 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.542815    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d2510e9511101f3c31e644d11e47ad8ac0210ca3b45db430d3d5ebaeb8b0ab1"} err="failed to get container status \"5d2510e9511101f3c31e644d11e47ad8ac0210ca3b45db430d3d5ebaeb8b0ab1\": rpc error: code = NotFound desc = could not find container \"5d2510e9511101f3c31e644d11e47ad8ac0210ca3b45db430d3d5ebaeb8b0ab1\": container with ID starting with 5d2510e9511101f3c31e644d11e47ad8ac0210ca3b45db430d3d5ebaeb8b0ab1 not found: ID does not exist"
Sep 18 13:44:34.578697 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.578670    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9"]
Sep 18 13:44:34.588000 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.587975    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-92b54f-9c4c54fbb-99vf9"]
Sep 18 13:44:34.632636 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.632572    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv"]
Sep 18 13:44:34.662168 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.662140    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-92b54f-7fc5d8c599-xmhhv"]
Sep 18 13:44:34.699459 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.699438    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92b54f-h24sm"]
Sep 18 13:44:34.704672 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.704642    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-92b54f-h24sm"]
Sep 18 13:44:34.850411 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.850379    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b873220-afcc-4185-ae47-d8b853b07971" path="/var/lib/kubelet/pods/2b873220-afcc-4185-ae47-d8b853b07971/volumes"
Sep 18 13:44:34.850797 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.850782    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64e08c40-2d69-494e-a0e3-aad00fb5fa6a" path="/var/lib/kubelet/pods/64e08c40-2d69-494e-a0e3-aad00fb5fa6a/volumes"
Sep 18 13:44:34.851129 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:34.851116    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcddd9ad-7afb-48b3-8aa2-61ef12ea0c97" path="/var/lib/kubelet/pods/dcddd9ad-7afb-48b3-8aa2-61ef12ea0c97/volumes"
Sep 18 13:44:52.777088 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:52.775548    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:44:59.419493 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:59.419455    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5be8a3-w8bhv"]
Sep 18 13:44:59.419875 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:59.419798    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcddd9ad-7afb-48b3-8aa2-61ef12ea0c97" containerName="revoke-keys"
Sep 18 13:44:59.419875 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:59.419809    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcddd9ad-7afb-48b3-8aa2-61ef12ea0c97" containerName="revoke-keys"
Sep 18 13:44:59.419875 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:59.419822    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b873220-afcc-4185-ae47-d8b853b07971" containerName="payload-processing"
Sep 18 13:44:59.419875 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:59.419828    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b873220-afcc-4185-ae47-d8b853b07971" containerName="payload-processing"
Sep 18 13:44:59.419875 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:59.419846    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64e08c40-2d69-494e-a0e3-aad00fb5fa6a" containerName="payload-pre-processing"
Sep 18 13:44:59.419875 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:59.419852    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="64e08c40-2d69-494e-a0e3-aad00fb5fa6a" containerName="payload-pre-processing"
Sep 18 13:44:59.419875 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:59.419860    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6de126ef-158b-46a2-ae50-8e664270679d" containerName="istio-proxy"
Sep 18 13:44:59.419875 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:59.419865    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6de126ef-158b-46a2-ae50-8e664270679d" containerName="istio-proxy"
Sep 18 13:44:59.420184 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:59.419937    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcddd9ad-7afb-48b3-8aa2-61ef12ea0c97" containerName="revoke-keys"
Sep 18 13:44:59.420184 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:59.419948    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="64e08c40-2d69-494e-a0e3-aad00fb5fa6a" containerName="payload-pre-processing"
Sep 18 13:44:59.420184 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:59.419956    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b873220-afcc-4185-ae47-d8b853b07971" containerName="payload-processing"
Sep 18 13:44:59.420184 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:59.419962    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6de126ef-158b-46a2-ae50-8e664270679d" containerName="istio-proxy"
Sep 18 13:44:59.422980 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:59.422960    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5be8a3-w8bhv"
Sep 18 13:44:59.425522 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:59.425493    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:44:59.426552 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:59.426534    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nh2cd\""
Sep 18 13:44:59.431884 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:59.431863    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5be8a3-w8bhv"]
Sep 18 13:44:59.467752 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:59.467719    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9735997-baa6-44bf-976b-b52fbb7d33ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5be8a3-w8bhv\" (UID: \"e9735997-baa6-44bf-976b-b52fbb7d33ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5be8a3-w8bhv"
Sep 18 13:44:59.568511 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:59.568477    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9735997-baa6-44bf-976b-b52fbb7d33ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5be8a3-w8bhv\" (UID: \"e9735997-baa6-44bf-976b-b52fbb7d33ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5be8a3-w8bhv"
Sep 18 13:44:59.569183 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:59.569159    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9735997-baa6-44bf-976b-b52fbb7d33ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5be8a3-w8bhv\" (UID: \"e9735997-baa6-44bf-976b-b52fbb7d33ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5be8a3-w8bhv"
Sep 18 13:44:59.734633 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:44:59.734543    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5be8a3-w8bhv"
Sep 18 13:45:00.024889 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:00.024862    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5be8a3-w8bhv"]
Sep 18 13:45:00.026442 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:45:00.025781    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9735997_baa6_44bf_976b_b52fbb7d33ca.slice/crio-38dfd53eab8749e8afe6a4869ee5d723c098b4097a37c301b7e72a5eb077064a WatchSource:0}: Error finding container 38dfd53eab8749e8afe6a4869ee5d723c098b4097a37c301b7e72a5eb077064a: Status 404 returned error can't find the container with id 38dfd53eab8749e8afe6a4869ee5d723c098b4097a37c301b7e72a5eb077064a
Sep 18 13:45:00.165914 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:00.165877    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828985-9fvm9"]
Sep 18 13:45:00.170691 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:00.170670    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828985-9fvm9"
Sep 18 13:45:00.180040 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:00.180022    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828985-9fvm9"
Sep 18 13:45:00.195715 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:00.193491    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828985-9fvm9"]
Sep 18 13:45:00.196841 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:00.196819    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-537abe-29828985-2rcvq"]
Sep 18 13:45:00.201382 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:00.201365    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-537abe-29828985-2rcvq"
Sep 18 13:45:00.214014 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:00.213990    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-537abe-29828985-2rcvq"
Sep 18 13:45:00.220866 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:00.220756    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-537abe-29828985-2rcvq"]
Sep 18 13:45:00.458935 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:00.458892    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29828985-9fvm9"]
Sep 18 13:45:00.462533 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:45:00.462507    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2424bf3_4834_4884_8c25_9f7f924769c9.slice/crio-2636ddf66e6469cb2de7e93e2668dec43b5ce42f027160d2e3697bf793b13c3d WatchSource:0}: Error finding container 2636ddf66e6469cb2de7e93e2668dec43b5ce42f027160d2e3697bf793b13c3d: Status 404 returned error can't find the container with id 2636ddf66e6469cb2de7e93e2668dec43b5ce42f027160d2e3697bf793b13c3d
Sep 18 13:45:00.502001 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:00.498687    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-537abe-29828985-2rcvq"]
Sep 18 13:45:00.660849 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:00.660815    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5be8a3-w8bhv" event={"ID":"e9735997-baa6-44bf-976b-b52fbb7d33ca","Type":"ContainerStarted","Data":"53c4670715f697dae3b772bb35d737f12dc9a0c677cb04df80f21ddb182e90e8"}
Sep 18 13:45:00.661011 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:00.660862    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5be8a3-w8bhv" event={"ID":"e9735997-baa6-44bf-976b-b52fbb7d33ca","Type":"ContainerStarted","Data":"38dfd53eab8749e8afe6a4869ee5d723c098b4097a37c301b7e72a5eb077064a"}
Sep 18 13:45:00.662084 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:00.662061    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-537abe-29828985-2rcvq" event={"ID":"b08413ad-899a-4368-bfe3-24c1385afb9a","Type":"ContainerStarted","Data":"e208d77246569c2390208a8b50d25b9acc2e6d4f3a3e3c444dff1ae1524e5914"}
Sep 18 13:45:00.663280 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:00.663259    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828985-9fvm9" event={"ID":"e2424bf3-4834-4884-8c25-9f7f924769c9","Type":"ContainerStarted","Data":"2636ddf66e6469cb2de7e93e2668dec43b5ce42f027160d2e3697bf793b13c3d"}
Sep 18 13:45:00.680208 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:00.680171    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5be8a3-w8bhv" podStartSLOduration=1.6801582590000002 podStartE2EDuration="1.680158259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:00.677560499 +0000 UTC m=+1798.033020349" watchObservedRunningTime="2026-09-18 13:45:00.680158259 +0000 UTC m=+1798.035618084"
Sep 18 13:45:01.670708 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:01.670626    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-537abe-29828985-2rcvq" event={"ID":"b08413ad-899a-4368-bfe3-24c1385afb9a","Type":"ContainerStarted","Data":"22cff72cd8cb97f5a9cf45b94ef685947ea6d96e9aab93fc76c737fbe3a2ea49"}
Sep 18 13:45:01.672364 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:01.672336    2586 generic.go:356] "Generic (PLEG): container finished" podID="e2424bf3-4834-4884-8c25-9f7f924769c9" containerID="96b4bfc878f09ba24311a693141830b2fa71f86995f0bdd3e674f3d59c4dab67" exitCode=0
Sep 18 13:45:01.672488 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:01.672411    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828985-9fvm9" event={"ID":"e2424bf3-4834-4884-8c25-9f7f924769c9","Type":"ContainerDied","Data":"96b4bfc878f09ba24311a693141830b2fa71f86995f0bdd3e674f3d59c4dab67"}
Sep 18 13:45:01.674151 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:01.674130    2586 generic.go:356] "Generic (PLEG): container finished" podID="e9735997-baa6-44bf-976b-b52fbb7d33ca" containerID="53c4670715f697dae3b772bb35d737f12dc9a0c677cb04df80f21ddb182e90e8" exitCode=0
Sep 18 13:45:01.674279 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:01.674215    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5be8a3-w8bhv" event={"ID":"e9735997-baa6-44bf-976b-b52fbb7d33ca","Type":"ContainerDied","Data":"53c4670715f697dae3b772bb35d737f12dc9a0c677cb04df80f21ddb182e90e8"}
Sep 18 13:45:01.697510 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:01.697470    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-537abe-29828985-2rcvq" podStartSLOduration=1.6974566439999998 podStartE2EDuration="1.697456644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:45:01.696202832 +0000 UTC m=+1799.051662657" watchObservedRunningTime="2026-09-18 13:45:01.697456644 +0000 UTC m=+1799.052916494"
Sep 18 13:45:02.681245 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:02.681210    2586 generic.go:356] "Generic (PLEG): container finished" podID="b08413ad-899a-4368-bfe3-24c1385afb9a" containerID="22cff72cd8cb97f5a9cf45b94ef685947ea6d96e9aab93fc76c737fbe3a2ea49" exitCode=0
Sep 18 13:45:02.681687 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:02.681299    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-537abe-29828985-2rcvq" event={"ID":"b08413ad-899a-4368-bfe3-24c1385afb9a","Type":"ContainerDied","Data":"22cff72cd8cb97f5a9cf45b94ef685947ea6d96e9aab93fc76c737fbe3a2ea49"}
Sep 18 13:45:02.813324 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:02.813292    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5be8a3-w8bhv"
Sep 18 13:45:02.855663 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:02.855640    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828985-9fvm9"
Sep 18 13:45:02.899163 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:02.899136    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9735997-baa6-44bf-976b-b52fbb7d33ca-maas-api-service-ca\") pod \"e9735997-baa6-44bf-976b-b52fbb7d33ca\" (UID: \"e9735997-baa6-44bf-976b-b52fbb7d33ca\") "
Sep 18 13:45:02.899488 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:02.899464    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9735997-baa6-44bf-976b-b52fbb7d33ca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e9735997-baa6-44bf-976b-b52fbb7d33ca" (UID: "e9735997-baa6-44bf-976b-b52fbb7d33ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:45:03.000551 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:03.000471    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9735997-baa6-44bf-976b-b52fbb7d33ca-maas-api-service-ca\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:45:03.688375 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:03.688345    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828985-9fvm9"
Sep 18 13:45:03.688813 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:03.688377    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29828985-9fvm9" event={"ID":"e2424bf3-4834-4884-8c25-9f7f924769c9","Type":"ContainerDied","Data":"2636ddf66e6469cb2de7e93e2668dec43b5ce42f027160d2e3697bf793b13c3d"}
Sep 18 13:45:03.688813 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:03.688426    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2636ddf66e6469cb2de7e93e2668dec43b5ce42f027160d2e3697bf793b13c3d"
Sep 18 13:45:03.689717 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:03.689695    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5be8a3-w8bhv"
Sep 18 13:45:03.689717 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:03.689713    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5be8a3-w8bhv" event={"ID":"e9735997-baa6-44bf-976b-b52fbb7d33ca","Type":"ContainerDied","Data":"38dfd53eab8749e8afe6a4869ee5d723c098b4097a37c301b7e72a5eb077064a"}
Sep 18 13:45:03.689954 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:03.689746    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38dfd53eab8749e8afe6a4869ee5d723c098b4097a37c301b7e72a5eb077064a"
Sep 18 13:45:03.825888 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:03.825862    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-537abe-29828985-2rcvq"
Sep 18 13:45:04.448087 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:04.448047    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"]
Sep 18 13:45:04.703657 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:04.703569    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-537abe-29828985-2rcvq"
Sep 18 13:45:04.703657 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:04.703581    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-537abe-29828985-2rcvq" event={"ID":"b08413ad-899a-4368-bfe3-24c1385afb9a","Type":"ContainerDied","Data":"e208d77246569c2390208a8b50d25b9acc2e6d4f3a3e3c444dff1ae1524e5914"}
Sep 18 13:45:04.704137 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:04.703668    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e208d77246569c2390208a8b50d25b9acc2e6d4f3a3e3c444dff1ae1524e5914"
Sep 18 13:45:04.937897 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:04.937872    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lcqcw_1cc32441-1189-4113-bf17-2769fa23a747/ovn-acl-logging/0.log"
Sep 18 13:45:04.940508 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:04.940490    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lcqcw_1cc32441-1189-4113-bf17-2769fa23a747/ovn-acl-logging/0.log"
Sep 18 13:45:05.087606 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:05.087529    2586 scope.go:117] "RemoveContainer" containerID="94d11ab2109a0c29479f594c9ecd967b5533bf351e37a70dc8f346995709e40b"
Sep 18 13:45:15.359300 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:15.359254    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d"]
Sep 18 13:45:15.359732 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:15.359517    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d" podUID="01b21c3d-49a1-4fb5-a925-71ee3ff2904f" containerName="payload-processing" containerID="cri-o://1e63a4dd7825c0d8d0617f0cb43af069be93a4f336757f4bf5cae79cd7a7e151" gracePeriod=30
Sep 18 13:45:15.586491 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:15.586459    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:45:15.621210 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:15.621157    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d"
Sep 18 13:45:15.715179 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:15.715152    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b21c3d-49a1-4fb5-a925-71ee3ff2904f-plugins-config-volume\") pod \"01b21c3d-49a1-4fb5-a925-71ee3ff2904f\" (UID: \"01b21c3d-49a1-4fb5-a925-71ee3ff2904f\") "
Sep 18 13:45:15.715361 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:15.715217    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lr9g8\" (UniqueName: \"kubernetes.io/projected/01b21c3d-49a1-4fb5-a925-71ee3ff2904f-kube-api-access-lr9g8\") pod \"01b21c3d-49a1-4fb5-a925-71ee3ff2904f\" (UID: \"01b21c3d-49a1-4fb5-a925-71ee3ff2904f\") "
Sep 18 13:45:15.715577 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:15.715553    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b21c3d-49a1-4fb5-a925-71ee3ff2904f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "01b21c3d-49a1-4fb5-a925-71ee3ff2904f" (UID: "01b21c3d-49a1-4fb5-a925-71ee3ff2904f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:45:15.717417 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:15.717394    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b21c3d-49a1-4fb5-a925-71ee3ff2904f-kube-api-access-lr9g8" (OuterVolumeSpecName: "kube-api-access-lr9g8") pod "01b21c3d-49a1-4fb5-a925-71ee3ff2904f" (UID: "01b21c3d-49a1-4fb5-a925-71ee3ff2904f"). InnerVolumeSpecName "kube-api-access-lr9g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:45:15.720804 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:15.720779    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5be8a3-w8bhv"]
Sep 18 13:45:15.724113 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:15.724095    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5be8a3-w8bhv"]
Sep 18 13:45:15.759677 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:15.759648    2586 generic.go:356] "Generic (PLEG): container finished" podID="01b21c3d-49a1-4fb5-a925-71ee3ff2904f" containerID="1e63a4dd7825c0d8d0617f0cb43af069be93a4f336757f4bf5cae79cd7a7e151" exitCode=0
Sep 18 13:45:15.759765 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:15.759705    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d"
Sep 18 13:45:15.759765 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:15.759737    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d" event={"ID":"01b21c3d-49a1-4fb5-a925-71ee3ff2904f","Type":"ContainerDied","Data":"1e63a4dd7825c0d8d0617f0cb43af069be93a4f336757f4bf5cae79cd7a7e151"}
Sep 18 13:45:15.759852 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:15.759787    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d" event={"ID":"01b21c3d-49a1-4fb5-a925-71ee3ff2904f","Type":"ContainerDied","Data":"6e2dfe88b126b7d4c65a40c9198168341fadc88bd1d35d7d700caec6c7cbb003"}
Sep 18 13:45:15.759852 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:15.759808    2586 scope.go:117] "RemoveContainer" containerID="1e63a4dd7825c0d8d0617f0cb43af069be93a4f336757f4bf5cae79cd7a7e151"
Sep 18 13:45:15.768646 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:15.768633    2586 scope.go:117] "RemoveContainer" containerID="1e63a4dd7825c0d8d0617f0cb43af069be93a4f336757f4bf5cae79cd7a7e151"
Sep 18 13:45:15.768930 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:45:15.768883    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e63a4dd7825c0d8d0617f0cb43af069be93a4f336757f4bf5cae79cd7a7e151\": container with ID starting with 1e63a4dd7825c0d8d0617f0cb43af069be93a4f336757f4bf5cae79cd7a7e151 not found: ID does not exist" containerID="1e63a4dd7825c0d8d0617f0cb43af069be93a4f336757f4bf5cae79cd7a7e151"
Sep 18 13:45:15.768990 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:15.768949    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e63a4dd7825c0d8d0617f0cb43af069be93a4f336757f4bf5cae79cd7a7e151"} err="failed to get container status \"1e63a4dd7825c0d8d0617f0cb43af069be93a4f336757f4bf5cae79cd7a7e151\": rpc error: code = NotFound desc = could not find container \"1e63a4dd7825c0d8d0617f0cb43af069be93a4f336757f4bf5cae79cd7a7e151\": container with ID starting with 1e63a4dd7825c0d8d0617f0cb43af069be93a4f336757f4bf5cae79cd7a7e151 not found: ID does not exist"
Sep 18 13:45:15.783120 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:15.783066    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d"]
Sep 18 13:45:15.785022 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:15.785001    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5be8a3-758f8bb47c-h6s6d"]
Sep 18 13:45:15.816890 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:15.816862    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01b21c3d-49a1-4fb5-a925-71ee3ff2904f-plugins-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:45:15.816890 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:15.816889    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr9g8\" (UniqueName: \"kubernetes.io/projected/01b21c3d-49a1-4fb5-a925-71ee3ff2904f-kube-api-access-lr9g8\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:45:16.849345 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:16.849309    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01b21c3d-49a1-4fb5-a925-71ee3ff2904f" path="/var/lib/kubelet/pods/01b21c3d-49a1-4fb5-a925-71ee3ff2904f/volumes"
Sep 18 13:45:16.849710 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:16.849649    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9735997-baa6-44bf-976b-b52fbb7d33ca" path="/var/lib/kubelet/pods/e9735997-baa6-44bf-976b-b52fbb7d33ca/volumes"
Sep 18 13:45:19.467921 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.467854    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2" podUID="9eb25909-07c3-49d0-ad0e-207793edcd72" containerName="main" containerID="cri-o://aee2d72746984ab542e0c013e5c0b98d8467f62b28ae32d852407bcf9fccb9db" gracePeriod=45
Sep 18 13:45:19.714158 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.714130    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:45:19.784649 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.784580    2586 generic.go:356] "Generic (PLEG): container finished" podID="9eb25909-07c3-49d0-ad0e-207793edcd72" containerID="aee2d72746984ab542e0c013e5c0b98d8467f62b28ae32d852407bcf9fccb9db" exitCode=0
Sep 18 13:45:19.784649 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.784618    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"
Sep 18 13:45:19.784832 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.784661    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2" event={"ID":"9eb25909-07c3-49d0-ad0e-207793edcd72","Type":"ContainerDied","Data":"aee2d72746984ab542e0c013e5c0b98d8467f62b28ae32d852407bcf9fccb9db"}
Sep 18 13:45:19.784832 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.784700    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2" event={"ID":"9eb25909-07c3-49d0-ad0e-207793edcd72","Type":"ContainerDied","Data":"0368d04514478da00ea5570bca7c4c7898c9f524a363497f4e883865a357f492"}
Sep 18 13:45:19.784832 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.784717    2586 scope.go:117] "RemoveContainer" containerID="aee2d72746984ab542e0c013e5c0b98d8467f62b28ae32d852407bcf9fccb9db"
Sep 18 13:45:19.794725 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.794694    2586 scope.go:117] "RemoveContainer" containerID="aee2d72746984ab542e0c013e5c0b98d8467f62b28ae32d852407bcf9fccb9db"
Sep 18 13:45:19.795165 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:45:19.795147    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aee2d72746984ab542e0c013e5c0b98d8467f62b28ae32d852407bcf9fccb9db\": container with ID starting with aee2d72746984ab542e0c013e5c0b98d8467f62b28ae32d852407bcf9fccb9db not found: ID does not exist" containerID="aee2d72746984ab542e0c013e5c0b98d8467f62b28ae32d852407bcf9fccb9db"
Sep 18 13:45:19.795235 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.795173    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aee2d72746984ab542e0c013e5c0b98d8467f62b28ae32d852407bcf9fccb9db"} err="failed to get container status \"aee2d72746984ab542e0c013e5c0b98d8467f62b28ae32d852407bcf9fccb9db\": rpc error: code = NotFound desc = could not find container \"aee2d72746984ab542e0c013e5c0b98d8467f62b28ae32d852407bcf9fccb9db\": container with ID starting with aee2d72746984ab542e0c013e5c0b98d8467f62b28ae32d852407bcf9fccb9db not found: ID does not exist"
Sep 18 13:45:19.855458 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.855428    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eb25909-07c3-49d0-ad0e-207793edcd72-model-cache\") pod \"9eb25909-07c3-49d0-ad0e-207793edcd72\" (UID: \"9eb25909-07c3-49d0-ad0e-207793edcd72\") "
Sep 18 13:45:19.855605 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.855479    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eb25909-07c3-49d0-ad0e-207793edcd72-tls-certs\") pod \"9eb25909-07c3-49d0-ad0e-207793edcd72\" (UID: \"9eb25909-07c3-49d0-ad0e-207793edcd72\") "
Sep 18 13:45:19.855605 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.855527    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eb25909-07c3-49d0-ad0e-207793edcd72-tmp-dir\") pod \"9eb25909-07c3-49d0-ad0e-207793edcd72\" (UID: \"9eb25909-07c3-49d0-ad0e-207793edcd72\") "
Sep 18 13:45:19.855605 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.855574    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eb25909-07c3-49d0-ad0e-207793edcd72-dshm\") pod \"9eb25909-07c3-49d0-ad0e-207793edcd72\" (UID: \"9eb25909-07c3-49d0-ad0e-207793edcd72\") "
Sep 18 13:45:19.855830 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.855644    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eb25909-07c3-49d0-ad0e-207793edcd72-home\") pod \"9eb25909-07c3-49d0-ad0e-207793edcd72\" (UID: \"9eb25909-07c3-49d0-ad0e-207793edcd72\") "
Sep 18 13:45:19.855830 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.855645    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb25909-07c3-49d0-ad0e-207793edcd72-model-cache" (OuterVolumeSpecName: "model-cache") pod "9eb25909-07c3-49d0-ad0e-207793edcd72" (UID: "9eb25909-07c3-49d0-ad0e-207793edcd72"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:45:19.855830 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.855708    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmg5k\" (UniqueName: \"kubernetes.io/projected/9eb25909-07c3-49d0-ad0e-207793edcd72-kube-api-access-bmg5k\") pod \"9eb25909-07c3-49d0-ad0e-207793edcd72\" (UID: \"9eb25909-07c3-49d0-ad0e-207793edcd72\") "
Sep 18 13:45:19.855830 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.855797    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb25909-07c3-49d0-ad0e-207793edcd72-home" (OuterVolumeSpecName: "home") pod "9eb25909-07c3-49d0-ad0e-207793edcd72" (UID: "9eb25909-07c3-49d0-ad0e-207793edcd72"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:45:19.856057 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.855834    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb25909-07c3-49d0-ad0e-207793edcd72-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9eb25909-07c3-49d0-ad0e-207793edcd72" (UID: "9eb25909-07c3-49d0-ad0e-207793edcd72"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:45:19.856057 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.856049    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eb25909-07c3-49d0-ad0e-207793edcd72-home\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:45:19.856140 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.856068    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eb25909-07c3-49d0-ad0e-207793edcd72-model-cache\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:45:19.856140 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.856082    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eb25909-07c3-49d0-ad0e-207793edcd72-tmp-dir\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:45:19.857808 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.857784    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb25909-07c3-49d0-ad0e-207793edcd72-kube-api-access-bmg5k" (OuterVolumeSpecName: "kube-api-access-bmg5k") pod "9eb25909-07c3-49d0-ad0e-207793edcd72" (UID: "9eb25909-07c3-49d0-ad0e-207793edcd72"). InnerVolumeSpecName "kube-api-access-bmg5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:45:19.857808 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.857799    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb25909-07c3-49d0-ad0e-207793edcd72-dshm" (OuterVolumeSpecName: "dshm") pod "9eb25909-07c3-49d0-ad0e-207793edcd72" (UID: "9eb25909-07c3-49d0-ad0e-207793edcd72"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:45:19.857987 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.857843    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eb25909-07c3-49d0-ad0e-207793edcd72-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9eb25909-07c3-49d0-ad0e-207793edcd72" (UID: "9eb25909-07c3-49d0-ad0e-207793edcd72"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:45:19.956656 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.956626    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eb25909-07c3-49d0-ad0e-207793edcd72-tls-certs\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:45:19.956656 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.956652    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eb25909-07c3-49d0-ad0e-207793edcd72-dshm\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:45:19.956656 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:19.956661    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmg5k\" (UniqueName: \"kubernetes.io/projected/9eb25909-07c3-49d0-ad0e-207793edcd72-kube-api-access-bmg5k\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:45:20.106475 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:20.106438    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"]
Sep 18 13:45:20.111800 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:20.111774    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-5be8a3/premium-w5-5be8a3-kserve-6456fd6c57-pccb2"]
Sep 18 13:45:20.849677 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:20.849642    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eb25909-07c3-49d0-ad0e-207793edcd72" path="/var/lib/kubelet/pods/9eb25909-07c3-49d0-ad0e-207793edcd72/volumes"
Sep 18 13:45:21.401709 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:21.401675    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"]
Sep 18 13:45:35.365451 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:35.365419    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:45:36.420890 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.420847    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq" podUID="a2a929d1-6dd1-4429-b1c9-a952ee5e71b6" containerName="main" containerID="cri-o://2d86d4752aeb88c15e28904a056fa7f1e7b5185dc300fd2c5b41a2232f1a802f" gracePeriod=45
Sep 18 13:45:36.671505 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.671453    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:45:36.803418 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.803387    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zd6bx\" (UniqueName: \"kubernetes.io/projected/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-kube-api-access-zd6bx\") pod \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\" (UID: \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\") "
Sep 18 13:45:36.803616 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.803440    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-tmp-dir\") pod \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\" (UID: \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\") "
Sep 18 13:45:36.803616 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.803496    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-tls-certs\") pod \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\" (UID: \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\") "
Sep 18 13:45:36.803616 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.803524    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-model-cache\") pod \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\" (UID: \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\") "
Sep 18 13:45:36.803616 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.803563    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-dshm\") pod \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\" (UID: \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\") "
Sep 18 13:45:36.803616 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.803598    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-home\") pod \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\" (UID: \"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6\") "
Sep 18 13:45:36.803933 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.803777    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2a929d1-6dd1-4429-b1c9-a952ee5e71b6" (UID: "a2a929d1-6dd1-4429-b1c9-a952ee5e71b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:45:36.804004 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.803945    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-tmp-dir\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:45:36.804062 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.804004    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-home" (OuterVolumeSpecName: "home") pod "a2a929d1-6dd1-4429-b1c9-a952ee5e71b6" (UID: "a2a929d1-6dd1-4429-b1c9-a952ee5e71b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:45:36.804120 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.804056    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2a929d1-6dd1-4429-b1c9-a952ee5e71b6" (UID: "a2a929d1-6dd1-4429-b1c9-a952ee5e71b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:45:36.806217 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.806189    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-dshm" (OuterVolumeSpecName: "dshm") pod "a2a929d1-6dd1-4429-b1c9-a952ee5e71b6" (UID: "a2a929d1-6dd1-4429-b1c9-a952ee5e71b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:45:36.806372 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.806351    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2a929d1-6dd1-4429-b1c9-a952ee5e71b6" (UID: "a2a929d1-6dd1-4429-b1c9-a952ee5e71b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:45:36.806436 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.806373    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-kube-api-access-zd6bx" (OuterVolumeSpecName: "kube-api-access-zd6bx") pod "a2a929d1-6dd1-4429-b1c9-a952ee5e71b6" (UID: "a2a929d1-6dd1-4429-b1c9-a952ee5e71b6"). InnerVolumeSpecName "kube-api-access-zd6bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:45:36.874896 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.874867    2586 generic.go:356] "Generic (PLEG): container finished" podID="a2a929d1-6dd1-4429-b1c9-a952ee5e71b6" containerID="2d86d4752aeb88c15e28904a056fa7f1e7b5185dc300fd2c5b41a2232f1a802f" exitCode=0
Sep 18 13:45:36.875028 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.874928    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"
Sep 18 13:45:36.875028 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.874948    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq" event={"ID":"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6","Type":"ContainerDied","Data":"2d86d4752aeb88c15e28904a056fa7f1e7b5185dc300fd2c5b41a2232f1a802f"}
Sep 18 13:45:36.875028 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.874988    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq" event={"ID":"a2a929d1-6dd1-4429-b1c9-a952ee5e71b6","Type":"ContainerDied","Data":"0f64334d14f4ba936be647beae06171f174a1890f747ab201bd25ccd53a37b73"}
Sep 18 13:45:36.875028 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.875004    2586 scope.go:117] "RemoveContainer" containerID="2d86d4752aeb88c15e28904a056fa7f1e7b5185dc300fd2c5b41a2232f1a802f"
Sep 18 13:45:36.883161 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.883139    2586 scope.go:117] "RemoveContainer" containerID="2d86d4752aeb88c15e28904a056fa7f1e7b5185dc300fd2c5b41a2232f1a802f"
Sep 18 13:45:36.883444 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:45:36.883427    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d86d4752aeb88c15e28904a056fa7f1e7b5185dc300fd2c5b41a2232f1a802f\": container with ID starting with 2d86d4752aeb88c15e28904a056fa7f1e7b5185dc300fd2c5b41a2232f1a802f not found: ID does not exist" containerID="2d86d4752aeb88c15e28904a056fa7f1e7b5185dc300fd2c5b41a2232f1a802f"
Sep 18 13:45:36.883516 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.883452    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d86d4752aeb88c15e28904a056fa7f1e7b5185dc300fd2c5b41a2232f1a802f"} err="failed to get container status \"2d86d4752aeb88c15e28904a056fa7f1e7b5185dc300fd2c5b41a2232f1a802f\": rpc error: code = NotFound desc = could not find container \"2d86d4752aeb88c15e28904a056fa7f1e7b5185dc300fd2c5b41a2232f1a802f\": container with ID starting with 2d86d4752aeb88c15e28904a056fa7f1e7b5185dc300fd2c5b41a2232f1a802f not found: ID does not exist"
Sep 18 13:45:36.893948 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.893891    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"]
Sep 18 13:45:36.898058 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.898038    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-537abe/auth-test-model-537abe-kserve-5d6fc8d7f-bqmbq"]
Sep 18 13:45:36.904764 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.904736    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd6bx\" (UniqueName: \"kubernetes.io/projected/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-kube-api-access-zd6bx\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:45:36.904867 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.904769    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-tls-certs\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:45:36.904867 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.904787    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-model-cache\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:45:36.904867 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.904799    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-dshm\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:45:36.904867 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:36.904811    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6-home\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:45:38.849025 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:38.848995    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2a929d1-6dd1-4429-b1c9-a952ee5e71b6" path="/var/lib/kubelet/pods/a2a929d1-6dd1-4429-b1c9-a952ee5e71b6/volumes"
Sep 18 13:45:44.580919 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:45:44.580873    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:46:03.447695 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.447659    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"]
Sep 18 13:46:03.448075 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.447997    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2424bf3-4834-4884-8c25-9f7f924769c9" containerName="cleanup"
Sep 18 13:46:03.448075 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.448008    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2424bf3-4834-4884-8c25-9f7f924769c9" containerName="cleanup"
Sep 18 13:46:03.448075 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.448019    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eb25909-07c3-49d0-ad0e-207793edcd72" containerName="main"
Sep 18 13:46:03.448075 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.448024    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eb25909-07c3-49d0-ad0e-207793edcd72" containerName="main"
Sep 18 13:46:03.448075 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.448032    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2a929d1-6dd1-4429-b1c9-a952ee5e71b6" containerName="main"
Sep 18 13:46:03.448075 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.448038    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2a929d1-6dd1-4429-b1c9-a952ee5e71b6" containerName="main"
Sep 18 13:46:03.448075 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.448049    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01b21c3d-49a1-4fb5-a925-71ee3ff2904f" containerName="payload-processing"
Sep 18 13:46:03.448075 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.448054    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="01b21c3d-49a1-4fb5-a925-71ee3ff2904f" containerName="payload-processing"
Sep 18 13:46:03.448075 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.448061    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9735997-baa6-44bf-976b-b52fbb7d33ca" containerName="revoke-keys"
Sep 18 13:46:03.448075 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.448067    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9735997-baa6-44bf-976b-b52fbb7d33ca" containerName="revoke-keys"
Sep 18 13:46:03.448075 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.448074    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b08413ad-899a-4368-bfe3-24c1385afb9a" containerName="cleanup"
Sep 18 13:46:03.448075 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.448079    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b08413ad-899a-4368-bfe3-24c1385afb9a" containerName="cleanup"
Sep 18 13:46:03.448433 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.448139    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eb25909-07c3-49d0-ad0e-207793edcd72" containerName="main"
Sep 18 13:46:03.448433 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.448149    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2a929d1-6dd1-4429-b1c9-a952ee5e71b6" containerName="main"
Sep 18 13:46:03.448433 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.448154    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="01b21c3d-49a1-4fb5-a925-71ee3ff2904f" containerName="payload-processing"
Sep 18 13:46:03.448433 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.448161    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b08413ad-899a-4368-bfe3-24c1385afb9a" containerName="cleanup"
Sep 18 13:46:03.448433 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.448168    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9735997-baa6-44bf-976b-b52fbb7d33ca" containerName="revoke-keys"
Sep 18 13:46:03.448433 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.448173    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2424bf3-4834-4884-8c25-9f7f924769c9" containerName="cleanup"
Sep 18 13:46:03.450859 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.450843    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:46:03.456430 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.456402    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c50e10\"/\"kube-root-ca.crt\""
Sep 18 13:46:03.457195 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.456970    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c50e10\"/\"default-dockercfg-g82ls\""
Sep 18 13:46:03.457291 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.457266    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c50e10\"/\"openshift-service-ca.crt\""
Sep 18 13:46:03.457394 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.457324    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c50e10\"/\"sub-test-model-c50e10-kserve-self-signed-certs\""
Sep 18 13:46:03.494335 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.494306    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"]
Sep 18 13:46:03.540011 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.539981    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aef98ea5-f364-4360-998a-1b1af114cab8-home\") pod \"sub-test-model-c50e10-kserve-c8c485f87-hxpt8\" (UID: \"aef98ea5-f364-4360-998a-1b1af114cab8\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:46:03.540245 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.540089    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aef98ea5-f364-4360-998a-1b1af114cab8-dshm\") pod \"sub-test-model-c50e10-kserve-c8c485f87-hxpt8\" (UID: \"aef98ea5-f364-4360-998a-1b1af114cab8\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:46:03.540245 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.540141    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbsft\" (UniqueName: \"kubernetes.io/projected/aef98ea5-f364-4360-998a-1b1af114cab8-kube-api-access-dbsft\") pod \"sub-test-model-c50e10-kserve-c8c485f87-hxpt8\" (UID: \"aef98ea5-f364-4360-998a-1b1af114cab8\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:46:03.540245 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.540223    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aef98ea5-f364-4360-998a-1b1af114cab8-model-cache\") pod \"sub-test-model-c50e10-kserve-c8c485f87-hxpt8\" (UID: \"aef98ea5-f364-4360-998a-1b1af114cab8\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:46:03.540431 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.540292    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aef98ea5-f364-4360-998a-1b1af114cab8-tmp-dir\") pod \"sub-test-model-c50e10-kserve-c8c485f87-hxpt8\" (UID: \"aef98ea5-f364-4360-998a-1b1af114cab8\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:46:03.540431 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.540359    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aef98ea5-f364-4360-998a-1b1af114cab8-tls-certs\") pod \"sub-test-model-c50e10-kserve-c8c485f87-hxpt8\" (UID: \"aef98ea5-f364-4360-998a-1b1af114cab8\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:46:03.641286 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.641257    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aef98ea5-f364-4360-998a-1b1af114cab8-dshm\") pod \"sub-test-model-c50e10-kserve-c8c485f87-hxpt8\" (UID: \"aef98ea5-f364-4360-998a-1b1af114cab8\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:46:03.641466 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.641298    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbsft\" (UniqueName: \"kubernetes.io/projected/aef98ea5-f364-4360-998a-1b1af114cab8-kube-api-access-dbsft\") pod \"sub-test-model-c50e10-kserve-c8c485f87-hxpt8\" (UID: \"aef98ea5-f364-4360-998a-1b1af114cab8\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:46:03.641466 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.641425    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aef98ea5-f364-4360-998a-1b1af114cab8-model-cache\") pod \"sub-test-model-c50e10-kserve-c8c485f87-hxpt8\" (UID: \"aef98ea5-f364-4360-998a-1b1af114cab8\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:46:03.641555 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.641476    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aef98ea5-f364-4360-998a-1b1af114cab8-tmp-dir\") pod \"sub-test-model-c50e10-kserve-c8c485f87-hxpt8\" (UID: \"aef98ea5-f364-4360-998a-1b1af114cab8\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:46:03.641555 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.641537    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aef98ea5-f364-4360-998a-1b1af114cab8-tls-certs\") pod \"sub-test-model-c50e10-kserve-c8c485f87-hxpt8\" (UID: \"aef98ea5-f364-4360-998a-1b1af114cab8\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:46:03.641643 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.641574    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aef98ea5-f364-4360-998a-1b1af114cab8-home\") pod \"sub-test-model-c50e10-kserve-c8c485f87-hxpt8\" (UID: \"aef98ea5-f364-4360-998a-1b1af114cab8\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:46:03.641848 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.641811    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aef98ea5-f364-4360-998a-1b1af114cab8-model-cache\") pod \"sub-test-model-c50e10-kserve-c8c485f87-hxpt8\" (UID: \"aef98ea5-f364-4360-998a-1b1af114cab8\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:46:03.641848 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.641828    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aef98ea5-f364-4360-998a-1b1af114cab8-tmp-dir\") pod \"sub-test-model-c50e10-kserve-c8c485f87-hxpt8\" (UID: \"aef98ea5-f364-4360-998a-1b1af114cab8\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:46:03.642052 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.641855    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aef98ea5-f364-4360-998a-1b1af114cab8-home\") pod \"sub-test-model-c50e10-kserve-c8c485f87-hxpt8\" (UID: \"aef98ea5-f364-4360-998a-1b1af114cab8\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:46:03.643731 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.643715    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aef98ea5-f364-4360-998a-1b1af114cab8-dshm\") pod \"sub-test-model-c50e10-kserve-c8c485f87-hxpt8\" (UID: \"aef98ea5-f364-4360-998a-1b1af114cab8\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:46:03.644004 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.643988    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aef98ea5-f364-4360-998a-1b1af114cab8-tls-certs\") pod \"sub-test-model-c50e10-kserve-c8c485f87-hxpt8\" (UID: \"aef98ea5-f364-4360-998a-1b1af114cab8\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:46:03.659506 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.659485    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbsft\" (UniqueName: \"kubernetes.io/projected/aef98ea5-f364-4360-998a-1b1af114cab8-kube-api-access-dbsft\") pod \"sub-test-model-c50e10-kserve-c8c485f87-hxpt8\" (UID: \"aef98ea5-f364-4360-998a-1b1af114cab8\") " pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:46:03.759613 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:03.759520    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:46:04.072665 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:04.072590    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"]
Sep 18 13:46:04.077474 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:46:04.077442    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaef98ea5_f364_4360_998a_1b1af114cab8.slice/crio-dbb50c1f43984b87a02bab7d935f9bb6d15f21c533a2b9f74d0ebfac7adec22a WatchSource:0}: Error finding container dbb50c1f43984b87a02bab7d935f9bb6d15f21c533a2b9f74d0ebfac7adec22a: Status 404 returned error can't find the container with id dbb50c1f43984b87a02bab7d935f9bb6d15f21c533a2b9f74d0ebfac7adec22a
Sep 18 13:46:05.011494 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:05.011458    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8" event={"ID":"aef98ea5-f364-4360-998a-1b1af114cab8","Type":"ContainerStarted","Data":"a7eb3ec832f303d4b9e9eaf60784ea86500a28ee5e4b762b9ebda5cc07a4ed41"}
Sep 18 13:46:05.011936 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:05.011504    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8" event={"ID":"aef98ea5-f364-4360-998a-1b1af114cab8","Type":"ContainerStarted","Data":"dbb50c1f43984b87a02bab7d935f9bb6d15f21c533a2b9f74d0ebfac7adec22a"}
Sep 18 13:46:05.034989 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:05.034951    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8" podStartSLOduration=2.034935767 podStartE2EDuration="2.034935767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:46:05.033475746 +0000 UTC m=+1862.388935552" watchObservedRunningTime="2026-09-18 13:46:05.034935767 +0000 UTC m=+1862.390395591"
Sep 18 13:46:07.279211 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:07.279173    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:46:13.759924 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:13.759865    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:46:13.760406 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:13.759934    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:46:13.771929 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:13.771888    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:46:14.067429 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:14.067332    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:46:21.010169 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:21.010088    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:46:33.689928 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:33.689870    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:46:44.150292 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:44.150258    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"]
Sep 18 13:46:47.781171 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:47.781125    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:46:48.796639 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:48.796600    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"]
Sep 18 13:46:48.800175 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:48.800157    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:46:48.803817 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:48.803784    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-537abe\"/\"default-dockercfg-vw4dd\""
Sep 18 13:46:48.804149 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:48.804127    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-537abe\"/\"openshift-service-ca.crt\""
Sep 18 13:46:48.804149 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:48.804138    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-537abe\"/\"kube-root-ca.crt\""
Sep 18 13:46:48.804664 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:48.804642    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-537abe\"/\"rate-test-model-537abe-kserve-self-signed-certs\""
Sep 18 13:46:48.811701 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:48.811679    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"]
Sep 18 13:46:48.866066 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:48.866027    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2186b3d9-da20-44bc-b842-eea400c2f297-home\") pod \"rate-test-model-537abe-kserve-867c8b9995-vkcdl\" (UID: \"2186b3d9-da20-44bc-b842-eea400c2f297\") " pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:46:48.866066 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:48.866067    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2186b3d9-da20-44bc-b842-eea400c2f297-model-cache\") pod \"rate-test-model-537abe-kserve-867c8b9995-vkcdl\" (UID: \"2186b3d9-da20-44bc-b842-eea400c2f297\") " pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:46:48.866314 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:48.866211    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2186b3d9-da20-44bc-b842-eea400c2f297-dshm\") pod \"rate-test-model-537abe-kserve-867c8b9995-vkcdl\" (UID: \"2186b3d9-da20-44bc-b842-eea400c2f297\") " pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:46:48.866314 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:48.866256    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pzbz\" (UniqueName: \"kubernetes.io/projected/2186b3d9-da20-44bc-b842-eea400c2f297-kube-api-access-5pzbz\") pod \"rate-test-model-537abe-kserve-867c8b9995-vkcdl\" (UID: \"2186b3d9-da20-44bc-b842-eea400c2f297\") " pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:46:48.866314 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:48.866296    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2186b3d9-da20-44bc-b842-eea400c2f297-tmp-dir\") pod \"rate-test-model-537abe-kserve-867c8b9995-vkcdl\" (UID: \"2186b3d9-da20-44bc-b842-eea400c2f297\") " pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:46:48.866452 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:48.866335    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2186b3d9-da20-44bc-b842-eea400c2f297-tls-certs\") pod \"rate-test-model-537abe-kserve-867c8b9995-vkcdl\" (UID: \"2186b3d9-da20-44bc-b842-eea400c2f297\") " pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:46:48.967161 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:48.967118    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2186b3d9-da20-44bc-b842-eea400c2f297-dshm\") pod \"rate-test-model-537abe-kserve-867c8b9995-vkcdl\" (UID: \"2186b3d9-da20-44bc-b842-eea400c2f297\") " pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:46:48.967161 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:48.967163    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pzbz\" (UniqueName: \"kubernetes.io/projected/2186b3d9-da20-44bc-b842-eea400c2f297-kube-api-access-5pzbz\") pod \"rate-test-model-537abe-kserve-867c8b9995-vkcdl\" (UID: \"2186b3d9-da20-44bc-b842-eea400c2f297\") " pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:46:48.967403 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:48.967289    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2186b3d9-da20-44bc-b842-eea400c2f297-tmp-dir\") pod \"rate-test-model-537abe-kserve-867c8b9995-vkcdl\" (UID: \"2186b3d9-da20-44bc-b842-eea400c2f297\") " pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:46:48.967403 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:48.967364    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2186b3d9-da20-44bc-b842-eea400c2f297-tls-certs\") pod \"rate-test-model-537abe-kserve-867c8b9995-vkcdl\" (UID: \"2186b3d9-da20-44bc-b842-eea400c2f297\") " pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:46:48.967521 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:48.967438    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2186b3d9-da20-44bc-b842-eea400c2f297-home\") pod \"rate-test-model-537abe-kserve-867c8b9995-vkcdl\" (UID: \"2186b3d9-da20-44bc-b842-eea400c2f297\") " pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:46:48.967521 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:48.967470    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2186b3d9-da20-44bc-b842-eea400c2f297-model-cache\") pod \"rate-test-model-537abe-kserve-867c8b9995-vkcdl\" (UID: \"2186b3d9-da20-44bc-b842-eea400c2f297\") " pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:46:48.967702 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:48.967682    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2186b3d9-da20-44bc-b842-eea400c2f297-tmp-dir\") pod \"rate-test-model-537abe-kserve-867c8b9995-vkcdl\" (UID: \"2186b3d9-da20-44bc-b842-eea400c2f297\") " pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:46:48.967771 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:48.967711    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2186b3d9-da20-44bc-b842-eea400c2f297-home\") pod \"rate-test-model-537abe-kserve-867c8b9995-vkcdl\" (UID: \"2186b3d9-da20-44bc-b842-eea400c2f297\") " pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:46:48.967811 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:48.967799    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2186b3d9-da20-44bc-b842-eea400c2f297-model-cache\") pod \"rate-test-model-537abe-kserve-867c8b9995-vkcdl\" (UID: \"2186b3d9-da20-44bc-b842-eea400c2f297\") " pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:46:48.969518 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:48.969493    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2186b3d9-da20-44bc-b842-eea400c2f297-dshm\") pod \"rate-test-model-537abe-kserve-867c8b9995-vkcdl\" (UID: \"2186b3d9-da20-44bc-b842-eea400c2f297\") " pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:46:48.969951 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:48.969936    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2186b3d9-da20-44bc-b842-eea400c2f297-tls-certs\") pod \"rate-test-model-537abe-kserve-867c8b9995-vkcdl\" (UID: \"2186b3d9-da20-44bc-b842-eea400c2f297\") " pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:46:48.977525 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:48.977505    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pzbz\" (UniqueName: \"kubernetes.io/projected/2186b3d9-da20-44bc-b842-eea400c2f297-kube-api-access-5pzbz\") pod \"rate-test-model-537abe-kserve-867c8b9995-vkcdl\" (UID: \"2186b3d9-da20-44bc-b842-eea400c2f297\") " pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:46:49.109925 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:49.109820    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:46:49.382723 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:49.382558    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"]
Sep 18 13:46:49.385407 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:46:49.385372    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2186b3d9_da20_44bc_b842_eea400c2f297.slice/crio-17ff774c518cdb94194b2c5edcb3d72502450111fa8033aac5a22c6954526c0c WatchSource:0}: Error finding container 17ff774c518cdb94194b2c5edcb3d72502450111fa8033aac5a22c6954526c0c: Status 404 returned error can't find the container with id 17ff774c518cdb94194b2c5edcb3d72502450111fa8033aac5a22c6954526c0c
Sep 18 13:46:50.243628 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:50.243594    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl" event={"ID":"2186b3d9-da20-44bc-b842-eea400c2f297","Type":"ContainerStarted","Data":"b42c46c9a223d5221f8ce863bde57cca106f99c7893ba292573125c88ce9479e"}
Sep 18 13:46:50.243628 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:50.243634    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl" event={"ID":"2186b3d9-da20-44bc-b842-eea400c2f297","Type":"ContainerStarted","Data":"17ff774c518cdb94194b2c5edcb3d72502450111fa8033aac5a22c6954526c0c"}
Sep 18 13:46:50.267107 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:50.267061    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl" podStartSLOduration=2.267044617 podStartE2EDuration="2.267044617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:46:50.26682716 +0000 UTC m=+1907.622286985" watchObservedRunningTime="2026-09-18 13:46:50.267044617 +0000 UTC m=+1907.622504455"
Sep 18 13:46:59.109960 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.109925    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:46:59.110424 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.109981    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:46:59.122553 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.122523    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:46:59.169891 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.169837    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8" podUID="aef98ea5-f364-4360-998a-1b1af114cab8" containerName="main" containerID="cri-o://a7eb3ec832f303d4b9e9eaf60784ea86500a28ee5e4b762b9ebda5cc07a4ed41" gracePeriod=45
Sep 18 13:46:59.288944 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.288879    2586 generic.go:356] "Generic (PLEG): container finished" podID="aef98ea5-f364-4360-998a-1b1af114cab8" containerID="a7eb3ec832f303d4b9e9eaf60784ea86500a28ee5e4b762b9ebda5cc07a4ed41" exitCode=0
Sep 18 13:46:59.289077 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.288951    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8" event={"ID":"aef98ea5-f364-4360-998a-1b1af114cab8","Type":"ContainerDied","Data":"a7eb3ec832f303d4b9e9eaf60784ea86500a28ee5e4b762b9ebda5cc07a4ed41"}
Sep 18 13:46:59.304789 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.304755    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:46:59.441342 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.441318    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:46:59.569643 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.569610    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aef98ea5-f364-4360-998a-1b1af114cab8-home\") pod \"aef98ea5-f364-4360-998a-1b1af114cab8\" (UID: \"aef98ea5-f364-4360-998a-1b1af114cab8\") "
Sep 18 13:46:59.569822 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.569690    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aef98ea5-f364-4360-998a-1b1af114cab8-tls-certs\") pod \"aef98ea5-f364-4360-998a-1b1af114cab8\" (UID: \"aef98ea5-f364-4360-998a-1b1af114cab8\") "
Sep 18 13:46:59.569822 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.569718    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aef98ea5-f364-4360-998a-1b1af114cab8-dshm\") pod \"aef98ea5-f364-4360-998a-1b1af114cab8\" (UID: \"aef98ea5-f364-4360-998a-1b1af114cab8\") "
Sep 18 13:46:59.569822 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.569748    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aef98ea5-f364-4360-998a-1b1af114cab8-model-cache\") pod \"aef98ea5-f364-4360-998a-1b1af114cab8\" (UID: \"aef98ea5-f364-4360-998a-1b1af114cab8\") "
Sep 18 13:46:59.569822 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.569814    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aef98ea5-f364-4360-998a-1b1af114cab8-tmp-dir\") pod \"aef98ea5-f364-4360-998a-1b1af114cab8\" (UID: \"aef98ea5-f364-4360-998a-1b1af114cab8\") "
Sep 18 13:46:59.570077 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.569847    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbsft\" (UniqueName: \"kubernetes.io/projected/aef98ea5-f364-4360-998a-1b1af114cab8-kube-api-access-dbsft\") pod \"aef98ea5-f364-4360-998a-1b1af114cab8\" (UID: \"aef98ea5-f364-4360-998a-1b1af114cab8\") "
Sep 18 13:46:59.570077 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.569873    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef98ea5-f364-4360-998a-1b1af114cab8-home" (OuterVolumeSpecName: "home") pod "aef98ea5-f364-4360-998a-1b1af114cab8" (UID: "aef98ea5-f364-4360-998a-1b1af114cab8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:46:59.570077 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.570055    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef98ea5-f364-4360-998a-1b1af114cab8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aef98ea5-f364-4360-998a-1b1af114cab8" (UID: "aef98ea5-f364-4360-998a-1b1af114cab8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:46:59.570207 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.570088    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef98ea5-f364-4360-998a-1b1af114cab8-model-cache" (OuterVolumeSpecName: "model-cache") pod "aef98ea5-f364-4360-998a-1b1af114cab8" (UID: "aef98ea5-f364-4360-998a-1b1af114cab8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:46:59.570207 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.570186    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aef98ea5-f364-4360-998a-1b1af114cab8-model-cache\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:46:59.570207 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.570198    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aef98ea5-f364-4360-998a-1b1af114cab8-tmp-dir\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:46:59.570207 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.570206    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aef98ea5-f364-4360-998a-1b1af114cab8-home\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:46:59.572238 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.572210    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aef98ea5-f364-4360-998a-1b1af114cab8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aef98ea5-f364-4360-998a-1b1af114cab8" (UID: "aef98ea5-f364-4360-998a-1b1af114cab8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:46:59.572358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.572245    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aef98ea5-f364-4360-998a-1b1af114cab8-kube-api-access-dbsft" (OuterVolumeSpecName: "kube-api-access-dbsft") pod "aef98ea5-f364-4360-998a-1b1af114cab8" (UID: "aef98ea5-f364-4360-998a-1b1af114cab8"). InnerVolumeSpecName "kube-api-access-dbsft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:46:59.572358 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.572274    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef98ea5-f364-4360-998a-1b1af114cab8-dshm" (OuterVolumeSpecName: "dshm") pod "aef98ea5-f364-4360-998a-1b1af114cab8" (UID: "aef98ea5-f364-4360-998a-1b1af114cab8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:46:59.592313 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.592287    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:46:59.671129 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.671107    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbsft\" (UniqueName: \"kubernetes.io/projected/aef98ea5-f364-4360-998a-1b1af114cab8-kube-api-access-dbsft\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:46:59.671129 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.671130    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aef98ea5-f364-4360-998a-1b1af114cab8-tls-certs\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:46:59.671129 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:46:59.671139    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aef98ea5-f364-4360-998a-1b1af114cab8-dshm\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:47:00.295100 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:00.295068    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"
Sep 18 13:47:00.295552 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:00.295108    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8" event={"ID":"aef98ea5-f364-4360-998a-1b1af114cab8","Type":"ContainerDied","Data":"dbb50c1f43984b87a02bab7d935f9bb6d15f21c533a2b9f74d0ebfac7adec22a"}
Sep 18 13:47:00.295552 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:00.295159    2586 scope.go:117] "RemoveContainer" containerID="a7eb3ec832f303d4b9e9eaf60784ea86500a28ee5e4b762b9ebda5cc07a4ed41"
Sep 18 13:47:00.317418 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:00.317391    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"]
Sep 18 13:47:00.322673 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:00.322646    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c50e10/sub-test-model-c50e10-kserve-c8c485f87-hxpt8"]
Sep 18 13:47:00.849493 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:00.849459    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aef98ea5-f364-4360-998a-1b1af114cab8" path="/var/lib/kubelet/pods/aef98ea5-f364-4360-998a-1b1af114cab8/volumes"
Sep 18 13:47:17.974590 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:17.974555    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:47:32.586458 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:32.586416    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:47:42.073719 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:42.073680    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:47:47.661860 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:47.661765    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"]
Sep 18 13:47:49.578100 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.578059    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"]
Sep 18 13:47:49.578531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.578406    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aef98ea5-f364-4360-998a-1b1af114cab8" containerName="main"
Sep 18 13:47:49.578531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.578417    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="aef98ea5-f364-4360-998a-1b1af114cab8" containerName="main"
Sep 18 13:47:49.578531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.578474    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="aef98ea5-f364-4360-998a-1b1af114cab8" containerName="main"
Sep 18 13:47:49.581471 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.581452    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.588457 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.588434    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2dd8de-openshift-default-dockercfg-7bvrm\""
Sep 18 13:47:49.637777 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.637734    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istio-envoy\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.637777 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.637779    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istio-data\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.638063 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.637806    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istio-podinfo\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.638063 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.637833    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istio-token\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.638063 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.637949    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-workload-socket\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.638063 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.637991    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-workload-certs\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.638063 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.638022    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qq5c\" (UniqueName: \"kubernetes.io/projected/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-kube-api-access-5qq5c\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.638246 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.638068    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-credential-socket\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.638246 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.638108    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istiod-ca-cert\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.686239 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.686203    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"]
Sep 18 13:47:49.739138 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.739098    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istio-envoy\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.739332 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.739147    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istio-data\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.739332 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.739180    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istio-podinfo\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.739332 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.739214    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istio-token\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.739332 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.739255    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-workload-socket\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.739332 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.739281    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-workload-certs\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.739332 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.739310    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qq5c\" (UniqueName: \"kubernetes.io/projected/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-kube-api-access-5qq5c\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.739711 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.739349    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-credential-socket\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.739711 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.739388    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istiod-ca-cert\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.739711 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.739592    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istio-data\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.739711 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.739671    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-workload-socket\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.740115 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.740093    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-workload-certs\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.740216 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.740198    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istiod-ca-cert\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.740448 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.740430    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-credential-socket\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.741598 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.741576    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istio-envoy\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.741873 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.741854    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istio-podinfo\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.752734 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.752709    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istio-token\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.753046 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.753025    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qq5c\" (UniqueName: \"kubernetes.io/projected/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-kube-api-access-5qq5c\") pod \"e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") " pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:49.891562 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:49.891529    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:50.195093 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:50.195057    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"]
Sep 18 13:47:50.196408 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:47:50.196382    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3cb21a3_3d69_4284_a76d_fa0a5a35d064.slice/crio-66a0d4157871300f2e24503b368e8a6c2ebe1df50e3562af142f7c8a1102718d WatchSource:0}: Error finding container 66a0d4157871300f2e24503b368e8a6c2ebe1df50e3562af142f7c8a1102718d: Status 404 returned error can't find the container with id 66a0d4157871300f2e24503b368e8a6c2ebe1df50e3562af142f7c8a1102718d
Sep 18 13:47:50.198765 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:50.198732    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:47:50.198853 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:50.198795    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:47:50.198853 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:50.198822    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:47:50.550536 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:50.550444    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t" event={"ID":"d3cb21a3-3d69-4284-a76d-fa0a5a35d064","Type":"ContainerStarted","Data":"a92e06f4dfc92da613b6a2e2a6a7718f8100fa3e8fe39b280f4c8944b653589e"}
Sep 18 13:47:50.550536 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:50.550503    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t" event={"ID":"d3cb21a3-3d69-4284-a76d-fa0a5a35d064","Type":"ContainerStarted","Data":"66a0d4157871300f2e24503b368e8a6c2ebe1df50e3562af142f7c8a1102718d"}
Sep 18 13:47:50.583564 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:50.583500    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t" podStartSLOduration=1.583478333 podStartE2EDuration="1.583478333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:47:50.580691843 +0000 UTC m=+1967.936151669" watchObservedRunningTime="2026-09-18 13:47:50.583478333 +0000 UTC m=+1967.938938159"
Sep 18 13:47:50.892096 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:50.892061    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:51.897284 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:51.897246    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:52.560839 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:52.560798    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:52.561838 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:52.561817    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:47:52.818440 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:52.818350    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:47:55.828185 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:55.828149    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q"]
Sep 18 13:47:55.831897 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:55.831877    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q"
Sep 18 13:47:55.834938 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:55.834892    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2dd8de-dockercfg-ddj6h\""
Sep 18 13:47:55.835075 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:55.834942    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2dd8de\""
Sep 18 13:47:55.857116 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:55.857081    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q"]
Sep 18 13:47:55.899519 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:55.899479    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js2nj\" (UniqueName: \"kubernetes.io/projected/2b1c489e-2a11-4289-b9fe-928cbf2e98ee-kube-api-access-js2nj\") pod \"payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q\" (UID: \"2b1c489e-2a11-4289-b9fe-928cbf2e98ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q"
Sep 18 13:47:55.899692 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:55.899624    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b1c489e-2a11-4289-b9fe-928cbf2e98ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q\" (UID: \"2b1c489e-2a11-4289-b9fe-928cbf2e98ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q"
Sep 18 13:47:56.000862 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:56.000805    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js2nj\" (UniqueName: \"kubernetes.io/projected/2b1c489e-2a11-4289-b9fe-928cbf2e98ee-kube-api-access-js2nj\") pod \"payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q\" (UID: \"2b1c489e-2a11-4289-b9fe-928cbf2e98ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q"
Sep 18 13:47:56.001103 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:56.000889    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b1c489e-2a11-4289-b9fe-928cbf2e98ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q\" (UID: \"2b1c489e-2a11-4289-b9fe-928cbf2e98ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q"
Sep 18 13:47:56.001961 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:56.001930    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b1c489e-2a11-4289-b9fe-928cbf2e98ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q\" (UID: \"2b1c489e-2a11-4289-b9fe-928cbf2e98ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q"
Sep 18 13:47:56.017154 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:56.017108    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js2nj\" (UniqueName: \"kubernetes.io/projected/2b1c489e-2a11-4289-b9fe-928cbf2e98ee-kube-api-access-js2nj\") pod \"payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q\" (UID: \"2b1c489e-2a11-4289-b9fe-928cbf2e98ee\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q"
Sep 18 13:47:56.142736 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:56.142695    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q"
Sep 18 13:47:56.422754 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:56.422724    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q"]
Sep 18 13:47:56.425428 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:47:56.425397    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1c489e_2a11_4289_b9fe_928cbf2e98ee.slice/crio-d37a9bc3a05717da7cb29261193db89b050d621a24923fc7f962edf7ca7400bb WatchSource:0}: Error finding container d37a9bc3a05717da7cb29261193db89b050d621a24923fc7f962edf7ca7400bb: Status 404 returned error can't find the container with id d37a9bc3a05717da7cb29261193db89b050d621a24923fc7f962edf7ca7400bb
Sep 18 13:47:56.582015 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:56.581975    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q" event={"ID":"2b1c489e-2a11-4289-b9fe-928cbf2e98ee","Type":"ContainerStarted","Data":"7265af0bb9332c5a793dcaa26cefd492d84f63a733d2e667ae2f84efa00f3ee8"}
Sep 18 13:47:56.582182 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:56.582038    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q" event={"ID":"2b1c489e-2a11-4289-b9fe-928cbf2e98ee","Type":"ContainerStarted","Data":"d37a9bc3a05717da7cb29261193db89b050d621a24923fc7f962edf7ca7400bb"}
Sep 18 13:47:56.582182 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:56.582157    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q"
Sep 18 13:47:56.614464 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:47:56.614409    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q" podStartSLOduration=1.614390764 podStartE2EDuration="1.614390764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:47:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:47:56.612024442 +0000 UTC m=+1973.967484267" watchObservedRunningTime="2026-09-18 13:47:56.614390764 +0000 UTC m=+1973.969850588"
Sep 18 13:48:02.681001 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:02.680933    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl" podUID="2186b3d9-da20-44bc-b842-eea400c2f297" containerName="main" containerID="cri-o://b42c46c9a223d5221f8ce863bde57cca106f99c7893ba292573125c88ce9479e" gracePeriod=45
Sep 18 13:48:02.980886 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:02.980861    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:48:03.071922 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:03.071870    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2186b3d9-da20-44bc-b842-eea400c2f297-dshm\") pod \"2186b3d9-da20-44bc-b842-eea400c2f297\" (UID: \"2186b3d9-da20-44bc-b842-eea400c2f297\") "
Sep 18 13:48:03.072121 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:03.071933    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2186b3d9-da20-44bc-b842-eea400c2f297-model-cache\") pod \"2186b3d9-da20-44bc-b842-eea400c2f297\" (UID: \"2186b3d9-da20-44bc-b842-eea400c2f297\") "
Sep 18 13:48:03.072121 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:03.071955    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2186b3d9-da20-44bc-b842-eea400c2f297-tls-certs\") pod \"2186b3d9-da20-44bc-b842-eea400c2f297\" (UID: \"2186b3d9-da20-44bc-b842-eea400c2f297\") "
Sep 18 13:48:03.072121 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:03.071984    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pzbz\" (UniqueName: \"kubernetes.io/projected/2186b3d9-da20-44bc-b842-eea400c2f297-kube-api-access-5pzbz\") pod \"2186b3d9-da20-44bc-b842-eea400c2f297\" (UID: \"2186b3d9-da20-44bc-b842-eea400c2f297\") "
Sep 18 13:48:03.072121 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:03.072007    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2186b3d9-da20-44bc-b842-eea400c2f297-tmp-dir\") pod \"2186b3d9-da20-44bc-b842-eea400c2f297\" (UID: \"2186b3d9-da20-44bc-b842-eea400c2f297\") "
Sep 18 13:48:03.072121 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:03.072046    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2186b3d9-da20-44bc-b842-eea400c2f297-home\") pod \"2186b3d9-da20-44bc-b842-eea400c2f297\" (UID: \"2186b3d9-da20-44bc-b842-eea400c2f297\") "
Sep 18 13:48:03.072391 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:03.072212    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2186b3d9-da20-44bc-b842-eea400c2f297-model-cache" (OuterVolumeSpecName: "model-cache") pod "2186b3d9-da20-44bc-b842-eea400c2f297" (UID: "2186b3d9-da20-44bc-b842-eea400c2f297"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:48:03.072391 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:03.072287    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2186b3d9-da20-44bc-b842-eea400c2f297-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2186b3d9-da20-44bc-b842-eea400c2f297" (UID: "2186b3d9-da20-44bc-b842-eea400c2f297"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:48:03.072391 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:03.072336    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2186b3d9-da20-44bc-b842-eea400c2f297-home" (OuterVolumeSpecName: "home") pod "2186b3d9-da20-44bc-b842-eea400c2f297" (UID: "2186b3d9-da20-44bc-b842-eea400c2f297"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:48:03.072549 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:03.072467    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2186b3d9-da20-44bc-b842-eea400c2f297-model-cache\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:48:03.072549 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:03.072482    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2186b3d9-da20-44bc-b842-eea400c2f297-tmp-dir\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:48:03.072549 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:03.072490    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2186b3d9-da20-44bc-b842-eea400c2f297-home\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:48:03.074328 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:03.074302    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2186b3d9-da20-44bc-b842-eea400c2f297-dshm" (OuterVolumeSpecName: "dshm") pod "2186b3d9-da20-44bc-b842-eea400c2f297" (UID: "2186b3d9-da20-44bc-b842-eea400c2f297"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:48:03.074439 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:03.074302    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2186b3d9-da20-44bc-b842-eea400c2f297-kube-api-access-5pzbz" (OuterVolumeSpecName: "kube-api-access-5pzbz") pod "2186b3d9-da20-44bc-b842-eea400c2f297" (UID: "2186b3d9-da20-44bc-b842-eea400c2f297"). InnerVolumeSpecName "kube-api-access-5pzbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:48:03.074439 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:03.074351    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2186b3d9-da20-44bc-b842-eea400c2f297-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2186b3d9-da20-44bc-b842-eea400c2f297" (UID: "2186b3d9-da20-44bc-b842-eea400c2f297"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:48:03.173219 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:03.173169    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2186b3d9-da20-44bc-b842-eea400c2f297-dshm\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:48:03.173219 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:03.173214    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2186b3d9-da20-44bc-b842-eea400c2f297-tls-certs\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:48:03.173219 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:03.173226    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pzbz\" (UniqueName: \"kubernetes.io/projected/2186b3d9-da20-44bc-b842-eea400c2f297-kube-api-access-5pzbz\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:48:03.625990 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:03.625945    2586 generic.go:356] "Generic (PLEG): container finished" podID="2186b3d9-da20-44bc-b842-eea400c2f297" containerID="b42c46c9a223d5221f8ce863bde57cca106f99c7893ba292573125c88ce9479e" exitCode=0
Sep 18 13:48:03.626210 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:03.626115    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl" event={"ID":"2186b3d9-da20-44bc-b842-eea400c2f297","Type":"ContainerDied","Data":"b42c46c9a223d5221f8ce863bde57cca106f99c7893ba292573125c88ce9479e"}
Sep 18 13:48:03.626210 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:03.626152    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl" event={"ID":"2186b3d9-da20-44bc-b842-eea400c2f297","Type":"ContainerDied","Data":"17ff774c518cdb94194b2c5edcb3d72502450111fa8033aac5a22c6954526c0c"}
Sep 18 13:48:03.626210 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:03.626175    2586 scope.go:117] "RemoveContainer" containerID="b42c46c9a223d5221f8ce863bde57cca106f99c7893ba292573125c88ce9479e"
Sep 18 13:48:03.626395 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:03.626360    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"
Sep 18 13:48:03.641376 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:03.641357    2586 scope.go:117] "RemoveContainer" containerID="b42c46c9a223d5221f8ce863bde57cca106f99c7893ba292573125c88ce9479e"
Sep 18 13:48:03.641678 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:48:03.641656    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b42c46c9a223d5221f8ce863bde57cca106f99c7893ba292573125c88ce9479e\": container with ID starting with b42c46c9a223d5221f8ce863bde57cca106f99c7893ba292573125c88ce9479e not found: ID does not exist" containerID="b42c46c9a223d5221f8ce863bde57cca106f99c7893ba292573125c88ce9479e"
Sep 18 13:48:03.641759 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:03.641688    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b42c46c9a223d5221f8ce863bde57cca106f99c7893ba292573125c88ce9479e"} err="failed to get container status \"b42c46c9a223d5221f8ce863bde57cca106f99c7893ba292573125c88ce9479e\": rpc error: code = NotFound desc = could not find container \"b42c46c9a223d5221f8ce863bde57cca106f99c7893ba292573125c88ce9479e\": container with ID starting with b42c46c9a223d5221f8ce863bde57cca106f99c7893ba292573125c88ce9479e not found: ID does not exist"
Sep 18 13:48:04.181527 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:04.181494    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"]
Sep 18 13:48:05.269368 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:05.269336    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-537abe/rate-test-model-537abe-kserve-867c8b9995-vkcdl"]
Sep 18 13:48:06.856598 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:06.856541    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2186b3d9-da20-44bc-b842-eea400c2f297" path="/var/lib/kubelet/pods/2186b3d9-da20-44bc-b842-eea400c2f297/volumes"
Sep 18 13:48:07.589717 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:07.589687    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q"
Sep 18 13:48:08.603046 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:08.603008    2586 patch_prober.go:28] interesting pod/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 18 13:48:08.605299 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:08.603061    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t" podUID="d3cb21a3-3d69-4284-a76d-fa0a5a35d064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 13:48:09.663400 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:09.663367    2586 generic.go:356] "Generic (PLEG): container finished" podID="d3cb21a3-3d69-4284-a76d-fa0a5a35d064" containerID="a92e06f4dfc92da613b6a2e2a6a7718f8100fa3e8fe39b280f4c8944b653589e" exitCode=0
Sep 18 13:48:09.663867 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:09.663445    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t" event={"ID":"d3cb21a3-3d69-4284-a76d-fa0a5a35d064","Type":"ContainerDied","Data":"a92e06f4dfc92da613b6a2e2a6a7718f8100fa3e8fe39b280f4c8944b653589e"}
Sep 18 13:48:09.663981 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:09.663962    2586 scope.go:117] "RemoveContainer" containerID="a92e06f4dfc92da613b6a2e2a6a7718f8100fa3e8fe39b280f4c8944b653589e"
Sep 18 13:48:09.665163 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:09.665134    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:48:09.665286 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:09.665218    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:48:09.665286 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:09.665265    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:48:10.670691 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:10.670653    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t" event={"ID":"d3cb21a3-3d69-4284-a76d-fa0a5a35d064","Type":"ContainerStarted","Data":"997ee940e13779cea5dedc256db7a16ebb7a9ec0f9948d771aaa973877a6ca42"}
Sep 18 13:48:10.891681 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:10.891645    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:48:10.891890 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:10.891863    2586 patch_prober.go:28] interesting pod/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 18 13:48:10.892008 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:10.891943    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t" podUID="d3cb21a3-3d69-4284-a76d-fa0a5a35d064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 18 13:48:11.589391 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:11.589360    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:48:11.892758 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:11.892723    2586 patch_prober.go:28] interesting pod/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 18 13:48:11.893223 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:11.892782    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t" podUID="d3cb21a3-3d69-4284-a76d-fa0a5a35d064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 18 13:48:12.892365 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:12.892329    2586 patch_prober.go:28] interesting pod/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 18 13:48:12.892535 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:12.892386    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t" podUID="d3cb21a3-3d69-4284-a76d-fa0a5a35d064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 18 13:48:13.892842 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:13.892805    2586 patch_prober.go:28] interesting pod/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 18 13:48:13.893276 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:13.892868    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t" podUID="d3cb21a3-3d69-4284-a76d-fa0a5a35d064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 18 13:48:14.892384 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:14.892348    2586 patch_prober.go:28] interesting pod/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 18 13:48:14.892564 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:14.892401    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t" podUID="d3cb21a3-3d69-4284-a76d-fa0a5a35d064" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 18 13:48:15.896058 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:15.896022    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:48:15.896458 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:15.896271    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:48:15.896875 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:15.896854    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:48:19.091587 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.091551    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"]
Sep 18 13:48:19.092165 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.092140    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2186b3d9-da20-44bc-b842-eea400c2f297" containerName="main"
Sep 18 13:48:19.092243 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.092169    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2186b3d9-da20-44bc-b842-eea400c2f297" containerName="main"
Sep 18 13:48:19.092313 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.092268    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2186b3d9-da20-44bc-b842-eea400c2f297" containerName="main"
Sep 18 13:48:19.095854 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.095833    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.099648 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.099630    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f629bd-openshift-default-dockercfg-pcvj2\""
Sep 18 13:48:19.118261 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.118225    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"]
Sep 18 13:48:19.227174 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.227135    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istio-data\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.227352 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.227179    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccmsg\" (UniqueName: \"kubernetes.io/projected/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-kube-api-access-ccmsg\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.227352 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.227210    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-workload-certs\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.227352 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.227297    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istio-token\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.227462 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.227369    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-credential-socket\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.227462 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.227398    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istio-envoy\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.227462 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.227422    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istio-podinfo\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.227551 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.227467    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-workload-socket\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.227592 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.227570    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istiod-ca-cert\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.328756 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.328726    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-credential-socket\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.328756 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.328758    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istio-envoy\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.329033 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.328774    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istio-podinfo\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.329033 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.328801    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-workload-socket\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.329033 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.328978    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istiod-ca-cert\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.329033 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.329018    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istio-data\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.329263 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.329041    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccmsg\" (UniqueName: \"kubernetes.io/projected/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-kube-api-access-ccmsg\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.329263 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.329074    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-workload-certs\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.329263 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.329132    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istio-token\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.329263 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.329153    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-credential-socket\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.329263 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.329251    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-workload-socket\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.329708 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.329323    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istio-data\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.329708 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.329453    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-workload-certs\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.329947 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.329886    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istiod-ca-cert\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.331377 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.331357    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istio-envoy\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.331596 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.331577    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istio-podinfo\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.339776 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.339755    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istio-token\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.339860 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.339822    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccmsg\" (UniqueName: \"kubernetes.io/projected/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-kube-api-access-ccmsg\") pod \"e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") " pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.407308 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.407278    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:19.703334 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.703187    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"]
Sep 18 13:48:19.705585 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:48:19.705551    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8cae6a_e1e8_4e4d_88fd_faa5eba60a65.slice/crio-77b9991b9256569ade994275ab29586db8b9d46d3f225a5a4ef6fdb0713e772b WatchSource:0}: Error finding container 77b9991b9256569ade994275ab29586db8b9d46d3f225a5a4ef6fdb0713e772b: Status 404 returned error can't find the container with id 77b9991b9256569ade994275ab29586db8b9d46d3f225a5a4ef6fdb0713e772b
Sep 18 13:48:19.708259 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.708220    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:48:19.708366 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.708305    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:48:19.708366 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.708345    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:48:19.720371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:19.720347    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh" event={"ID":"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65","Type":"ContainerStarted","Data":"77b9991b9256569ade994275ab29586db8b9d46d3f225a5a4ef6fdb0713e772b"}
Sep 18 13:48:20.727057 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:20.727013    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh" event={"ID":"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65","Type":"ContainerStarted","Data":"ddb662268bdc55bb5f78278232492f940e86bd275f1098b37c6395d9bedabdaf"}
Sep 18 13:48:21.407728 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:21.407688    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:21.412502 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:21.412476    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:21.455426 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:21.455379    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh" podStartSLOduration=2.455365973 podStartE2EDuration="2.455365973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:48:20.77380877 +0000 UTC m=+1998.129268595" watchObservedRunningTime="2026-09-18 13:48:21.455365973 +0000 UTC m=+1998.810825798"
Sep 18 13:48:21.732537 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:21.732453    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:21.733419 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:21.733389    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:32.421982 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:32.421945    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:48:41.835511 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:41.835479    2586 generic.go:356] "Generic (PLEG): container finished" podID="2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65" containerID="ddb662268bdc55bb5f78278232492f940e86bd275f1098b37c6395d9bedabdaf" exitCode=0
Sep 18 13:48:41.836046 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:41.835554    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh" event={"ID":"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65","Type":"ContainerDied","Data":"ddb662268bdc55bb5f78278232492f940e86bd275f1098b37c6395d9bedabdaf"}
Sep 18 13:48:41.836120 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:41.836077    2586 scope.go:117] "RemoveContainer" containerID="ddb662268bdc55bb5f78278232492f940e86bd275f1098b37c6395d9bedabdaf"
Sep 18 13:48:41.837182 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:41.837155    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:48:41.837244 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:41.837221    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:48:41.837283 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:41.837250    2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 18 13:48:42.842376 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:42.842341    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh" event={"ID":"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65","Type":"ContainerStarted","Data":"e9bed24d4edf2f234cd05a52cd5e38934c725f5356dd8ee6024d7b7f6d133d4d"}
Sep 18 13:48:43.407989 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:43.407951    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:43.408221 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:43.408188    2586 patch_prober.go:28] interesting pod/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 18 13:48:43.408301 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:43.408252    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh" podUID="2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 18 13:48:44.408577 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:44.408531    2586 patch_prober.go:28] interesting pod/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 18 13:48:44.409089 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:44.408608    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh" podUID="2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 18 13:48:45.420930 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:45.419869    2586 patch_prober.go:28] interesting pod/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 18 13:48:45.420930 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:45.419984    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh" podUID="2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 18 13:48:46.408600 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:46.408559    2586 patch_prober.go:28] interesting pod/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 18 13:48:46.408887 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:46.408855    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh" podUID="2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 18 13:48:47.407854 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:47.407814    2586 patch_prober.go:28] interesting pod/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 18 13:48:47.408454 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:47.407929    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh" podUID="2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 18 13:48:48.411681 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:48.411647    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:48.412162 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:48.411929    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:48:48.412725 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:48:48.412705    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:49:03.380183 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.380147    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"]
Sep 18 13:49:03.382683 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.382660    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:49:03.385049 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.385018    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2dd8de\"/\"kube-root-ca.crt\""
Sep 18 13:49:03.386063 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.386031    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2dd8de\"/\"default-dockercfg-2gb2h\""
Sep 18 13:49:03.386265 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.386240    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2dd8de\"/\"openshift-service-ca.crt\""
Sep 18 13:49:03.386365 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.386343    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2dd8de\"/\"ipp-route-2dd8de-kserve-self-signed-certs\""
Sep 18 13:49:03.397281 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.397252    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"]
Sep 18 13:49:03.539792 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.539759    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2dbfe68-4397-40b0-bee5-fc4912111813-home\") pod \"ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj\" (UID: \"c2dbfe68-4397-40b0-bee5-fc4912111813\") " pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:49:03.539996 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.539812    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s829h\" (UniqueName: \"kubernetes.io/projected/c2dbfe68-4397-40b0-bee5-fc4912111813-kube-api-access-s829h\") pod \"ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj\" (UID: \"c2dbfe68-4397-40b0-bee5-fc4912111813\") " pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:49:03.539996 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.539844    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2dbfe68-4397-40b0-bee5-fc4912111813-tls-certs\") pod \"ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj\" (UID: \"c2dbfe68-4397-40b0-bee5-fc4912111813\") " pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:49:03.539996 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.539877    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2dbfe68-4397-40b0-bee5-fc4912111813-tmp-dir\") pod \"ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj\" (UID: \"c2dbfe68-4397-40b0-bee5-fc4912111813\") " pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:49:03.539996 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.539934    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2dbfe68-4397-40b0-bee5-fc4912111813-model-cache\") pod \"ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj\" (UID: \"c2dbfe68-4397-40b0-bee5-fc4912111813\") " pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:49:03.539996 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.539952    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2dbfe68-4397-40b0-bee5-fc4912111813-dshm\") pod \"ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj\" (UID: \"c2dbfe68-4397-40b0-bee5-fc4912111813\") " pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:49:03.640752 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.640717    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s829h\" (UniqueName: \"kubernetes.io/projected/c2dbfe68-4397-40b0-bee5-fc4912111813-kube-api-access-s829h\") pod \"ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj\" (UID: \"c2dbfe68-4397-40b0-bee5-fc4912111813\") " pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:49:03.640963 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.640770    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2dbfe68-4397-40b0-bee5-fc4912111813-tls-certs\") pod \"ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj\" (UID: \"c2dbfe68-4397-40b0-bee5-fc4912111813\") " pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:49:03.640963 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.640788    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2dbfe68-4397-40b0-bee5-fc4912111813-tmp-dir\") pod \"ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj\" (UID: \"c2dbfe68-4397-40b0-bee5-fc4912111813\") " pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:49:03.640963 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.640810    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2dbfe68-4397-40b0-bee5-fc4912111813-model-cache\") pod \"ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj\" (UID: \"c2dbfe68-4397-40b0-bee5-fc4912111813\") " pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:49:03.640963 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.640829    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2dbfe68-4397-40b0-bee5-fc4912111813-dshm\") pod \"ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj\" (UID: \"c2dbfe68-4397-40b0-bee5-fc4912111813\") " pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:49:03.640963 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.640952    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2dbfe68-4397-40b0-bee5-fc4912111813-home\") pod \"ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj\" (UID: \"c2dbfe68-4397-40b0-bee5-fc4912111813\") " pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:49:03.641280 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.641256    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2dbfe68-4397-40b0-bee5-fc4912111813-tmp-dir\") pod \"ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj\" (UID: \"c2dbfe68-4397-40b0-bee5-fc4912111813\") " pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:49:03.641333 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.641291    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2dbfe68-4397-40b0-bee5-fc4912111813-model-cache\") pod \"ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj\" (UID: \"c2dbfe68-4397-40b0-bee5-fc4912111813\") " pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:49:03.641333 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.641308    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2dbfe68-4397-40b0-bee5-fc4912111813-home\") pod \"ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj\" (UID: \"c2dbfe68-4397-40b0-bee5-fc4912111813\") " pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:49:03.643377 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.643360    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2dbfe68-4397-40b0-bee5-fc4912111813-dshm\") pod \"ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj\" (UID: \"c2dbfe68-4397-40b0-bee5-fc4912111813\") " pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:49:03.643558 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.643542    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2dbfe68-4397-40b0-bee5-fc4912111813-tls-certs\") pod \"ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj\" (UID: \"c2dbfe68-4397-40b0-bee5-fc4912111813\") " pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:49:03.649308 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.649287    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s829h\" (UniqueName: \"kubernetes.io/projected/c2dbfe68-4397-40b0-bee5-fc4912111813-kube-api-access-s829h\") pod \"ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj\" (UID: \"c2dbfe68-4397-40b0-bee5-fc4912111813\") " pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:49:03.697490 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.697460    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:49:03.972092 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:03.972069    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"]
Sep 18 13:49:03.973997 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:49:03.973967    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2dbfe68_4397_40b0_bee5_fc4912111813.slice/crio-8809e62c7ca3ebf122eb592c694486646e194471cdfb6a6a9bc6aecb18fbf2f4 WatchSource:0}: Error finding container 8809e62c7ca3ebf122eb592c694486646e194471cdfb6a6a9bc6aecb18fbf2f4: Status 404 returned error can't find the container with id 8809e62c7ca3ebf122eb592c694486646e194471cdfb6a6a9bc6aecb18fbf2f4
Sep 18 13:49:04.956964 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:04.956930    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj" event={"ID":"c2dbfe68-4397-40b0-bee5-fc4912111813","Type":"ContainerStarted","Data":"edbe80ed86a04c5d7b9a4dbf7156afd8573e03587b99d603287b5e74aaf285ef"}
Sep 18 13:49:04.956964 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:04.956967    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj" event={"ID":"c2dbfe68-4397-40b0-bee5-fc4912111813","Type":"ContainerStarted","Data":"8809e62c7ca3ebf122eb592c694486646e194471cdfb6a6a9bc6aecb18fbf2f4"}
Sep 18 13:49:04.982019 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:04.981974    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj" podStartSLOduration=1.981959507 podStartE2EDuration="1.981959507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:49:04.979956053 +0000 UTC m=+2042.335415877" watchObservedRunningTime="2026-09-18 13:49:04.981959507 +0000 UTC m=+2042.337419335"
Sep 18 13:49:07.384180 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:07.384142    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:49:13.698161 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:13.698123    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:49:13.698616 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:13.698379    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:49:13.711363 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:13.711341    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:49:14.020362 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:14.020280    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:49:23.374016 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:23.373923    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:49:39.848270 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:39.848237    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p"]
Sep 18 13:49:39.850957 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:39.850938    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p"
Sep 18 13:49:39.854169 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:39.854150    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-eaff22-dockercfg-ltk9q\""
Sep 18 13:49:39.854662 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:39.854644    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-eaff22\""
Sep 18 13:49:39.864372 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:39.864347    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p"]
Sep 18 13:49:39.960249 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:39.960214    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6"]
Sep 18 13:49:39.963219 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:39.963200    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6"
Sep 18 13:49:39.971755 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:39.971733    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01fef123-0cf2-481a-92c0-67d27d09d812-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p\" (UID: \"01fef123-0cf2-481a-92c0-67d27d09d812\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p"
Sep 18 13:49:39.971870 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:39.971855    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlhg8\" (UniqueName: \"kubernetes.io/projected/01fef123-0cf2-481a-92c0-67d27d09d812-kube-api-access-dlhg8\") pod \"payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p\" (UID: \"01fef123-0cf2-481a-92c0-67d27d09d812\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p"
Sep 18 13:49:39.986013 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:39.985987    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6"]
Sep 18 13:49:40.072469 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:40.072440    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c07c3d25-46c8-4d16-a94e-1b56c8c4faa9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6\" (UID: \"c07c3d25-46c8-4d16-a94e-1b56c8c4faa9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6"
Sep 18 13:49:40.072625 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:40.072481    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlhg8\" (UniqueName: \"kubernetes.io/projected/01fef123-0cf2-481a-92c0-67d27d09d812-kube-api-access-dlhg8\") pod \"payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p\" (UID: \"01fef123-0cf2-481a-92c0-67d27d09d812\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p"
Sep 18 13:49:40.072625 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:40.072504    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01fef123-0cf2-481a-92c0-67d27d09d812-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p\" (UID: \"01fef123-0cf2-481a-92c0-67d27d09d812\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p"
Sep 18 13:49:40.072625 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:40.072534    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd8r6\" (UniqueName: \"kubernetes.io/projected/c07c3d25-46c8-4d16-a94e-1b56c8c4faa9-kube-api-access-sd8r6\") pod \"payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6\" (UID: \"c07c3d25-46c8-4d16-a94e-1b56c8c4faa9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6"
Sep 18 13:49:40.073170 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:40.073152    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01fef123-0cf2-481a-92c0-67d27d09d812-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p\" (UID: \"01fef123-0cf2-481a-92c0-67d27d09d812\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p"
Sep 18 13:49:40.090966 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:40.090938    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlhg8\" (UniqueName: \"kubernetes.io/projected/01fef123-0cf2-481a-92c0-67d27d09d812-kube-api-access-dlhg8\") pod \"payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p\" (UID: \"01fef123-0cf2-481a-92c0-67d27d09d812\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p"
Sep 18 13:49:40.161374 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:40.161349    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p"
Sep 18 13:49:40.173476 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:40.173442    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd8r6\" (UniqueName: \"kubernetes.io/projected/c07c3d25-46c8-4d16-a94e-1b56c8c4faa9-kube-api-access-sd8r6\") pod \"payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6\" (UID: \"c07c3d25-46c8-4d16-a94e-1b56c8c4faa9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6"
Sep 18 13:49:40.173564 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:40.173546    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c07c3d25-46c8-4d16-a94e-1b56c8c4faa9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6\" (UID: \"c07c3d25-46c8-4d16-a94e-1b56c8c4faa9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6"
Sep 18 13:49:40.174167 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:40.174141    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c07c3d25-46c8-4d16-a94e-1b56c8c4faa9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6\" (UID: \"c07c3d25-46c8-4d16-a94e-1b56c8c4faa9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6"
Sep 18 13:49:40.194249 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:40.194223    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd8r6\" (UniqueName: \"kubernetes.io/projected/c07c3d25-46c8-4d16-a94e-1b56c8c4faa9-kube-api-access-sd8r6\") pod \"payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6\" (UID: \"c07c3d25-46c8-4d16-a94e-1b56c8c4faa9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6"
Sep 18 13:49:40.273180 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:40.273153    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6"
Sep 18 13:49:40.445671 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:40.445648    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p"]
Sep 18 13:49:40.447292 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:49:40.447265    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01fef123_0cf2_481a_92c0_67d27d09d812.slice/crio-e313a1e04f044c066abac6a5e56e5a2eb88e73e2dd6bb57822ad624352c5e247 WatchSource:0}: Error finding container e313a1e04f044c066abac6a5e56e5a2eb88e73e2dd6bb57822ad624352c5e247: Status 404 returned error can't find the container with id e313a1e04f044c066abac6a5e56e5a2eb88e73e2dd6bb57822ad624352c5e247
Sep 18 13:49:40.563453 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:40.563416    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6"]
Sep 18 13:49:40.566167 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:49:40.566139    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07c3d25_46c8_4d16_a94e_1b56c8c4faa9.slice/crio-70bdc88a6c60e47360b436874d2645e044efb4d856c195bd16b4a424d11b16bb WatchSource:0}: Error finding container 70bdc88a6c60e47360b436874d2645e044efb4d856c195bd16b4a424d11b16bb: Status 404 returned error can't find the container with id 70bdc88a6c60e47360b436874d2645e044efb4d856c195bd16b4a424d11b16bb
Sep 18 13:49:41.143505 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:41.143454    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p" event={"ID":"01fef123-0cf2-481a-92c0-67d27d09d812","Type":"ContainerStarted","Data":"fa3907f3fdf8b8633148a9a603e73dfb7557f0a4e1c92fa805f5cfe3bfba6478"}
Sep 18 13:49:41.143951 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:41.143522    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p" event={"ID":"01fef123-0cf2-481a-92c0-67d27d09d812","Type":"ContainerStarted","Data":"e313a1e04f044c066abac6a5e56e5a2eb88e73e2dd6bb57822ad624352c5e247"}
Sep 18 13:49:41.143951 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:41.143619    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p"
Sep 18 13:49:41.145061 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:41.145038    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6" event={"ID":"c07c3d25-46c8-4d16-a94e-1b56c8c4faa9","Type":"ContainerStarted","Data":"bbde2c73880a3dd5848bbdebee9f192487bb19b8e17d11e72e2056bcbf57b44c"}
Sep 18 13:49:41.145176 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:41.145087    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6" event={"ID":"c07c3d25-46c8-4d16-a94e-1b56c8c4faa9","Type":"ContainerStarted","Data":"70bdc88a6c60e47360b436874d2645e044efb4d856c195bd16b4a424d11b16bb"}
Sep 18 13:49:41.145176 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:41.145139    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6"
Sep 18 13:49:41.228489 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:41.228438    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p" podStartSLOduration=2.228421853 podStartE2EDuration="2.228421853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:49:41.189771085 +0000 UTC m=+2078.545230907" watchObservedRunningTime="2026-09-18 13:49:41.228421853 +0000 UTC m=+2078.583881679"
Sep 18 13:49:41.228651 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:41.228537    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6" podStartSLOduration=2.228530766 podStartE2EDuration="2.228530766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:49:41.221337822 +0000 UTC m=+2078.576797657" watchObservedRunningTime="2026-09-18 13:49:41.228530766 +0000 UTC m=+2078.583990593"
Sep 18 13:49:43.798538 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:43.798500    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:49:52.152717 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:52.152689    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p"
Sep 18 13:49:52.153266 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:49:52.152755    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6"
Sep 18 13:50:04.975873 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:04.975839    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lcqcw_1cc32441-1189-4113-bf17-2769fa23a747/ovn-acl-logging/0.log"
Sep 18 13:50:04.979443 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:04.979421    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lcqcw_1cc32441-1189-4113-bf17-2769fa23a747/ovn-acl-logging/0.log"
Sep 18 13:50:05.186490 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:05.186459    2586 scope.go:117] "RemoveContainer" containerID="dbaba5c5c32711b432d5ddd0e14079444e2197d6947272649da4a3c9bb785baa"
Sep 18 13:50:05.194881 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:05.194862    2586 scope.go:117] "RemoveContainer" containerID="0f4e53b6a18d433e9f81b4c3d5fe4612d04e3293b317ea219227ad7ff1a04fc8"
Sep 18 13:50:16.032828 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:16.032794    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6"]
Sep 18 13:50:16.033313 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:16.033035    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6" podUID="c07c3d25-46c8-4d16-a94e-1b56c8c4faa9" containerName="payload-processing" containerID="cri-o://bbde2c73880a3dd5848bbdebee9f192487bb19b8e17d11e72e2056bcbf57b44c" gracePeriod=30
Sep 18 13:50:16.060589 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:16.060552    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p"]
Sep 18 13:50:16.060827 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:16.060798    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p" podUID="01fef123-0cf2-481a-92c0-67d27d09d812" containerName="payload-pre-processing" containerID="cri-o://fa3907f3fdf8b8633148a9a603e73dfb7557f0a4e1c92fa805f5cfe3bfba6478" gracePeriod=30
Sep 18 13:50:16.323175 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:16.323122    2586 generic.go:356] "Generic (PLEG): container finished" podID="c07c3d25-46c8-4d16-a94e-1b56c8c4faa9" containerID="bbde2c73880a3dd5848bbdebee9f192487bb19b8e17d11e72e2056bcbf57b44c" exitCode=0
Sep 18 13:50:16.323175 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:16.323142    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6" event={"ID":"c07c3d25-46c8-4d16-a94e-1b56c8c4faa9","Type":"ContainerDied","Data":"bbde2c73880a3dd5848bbdebee9f192487bb19b8e17d11e72e2056bcbf57b44c"}
Sep 18 13:50:16.325055 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:16.325034    2586 generic.go:356] "Generic (PLEG): container finished" podID="01fef123-0cf2-481a-92c0-67d27d09d812" containerID="fa3907f3fdf8b8633148a9a603e73dfb7557f0a4e1c92fa805f5cfe3bfba6478" exitCode=0
Sep 18 13:50:16.325166 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:16.325105    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p" event={"ID":"01fef123-0cf2-481a-92c0-67d27d09d812","Type":"ContainerDied","Data":"fa3907f3fdf8b8633148a9a603e73dfb7557f0a4e1c92fa805f5cfe3bfba6478"}
Sep 18 13:50:16.345767 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:16.345749    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6"
Sep 18 13:50:16.348609 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:16.348588    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p"
Sep 18 13:50:16.421016 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:16.420988    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01fef123-0cf2-481a-92c0-67d27d09d812-plugins-config-volume\") pod \"01fef123-0cf2-481a-92c0-67d27d09d812\" (UID: \"01fef123-0cf2-481a-92c0-67d27d09d812\") "
Sep 18 13:50:16.421180 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:16.421041    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c07c3d25-46c8-4d16-a94e-1b56c8c4faa9-plugins-config-volume\") pod \"c07c3d25-46c8-4d16-a94e-1b56c8c4faa9\" (UID: \"c07c3d25-46c8-4d16-a94e-1b56c8c4faa9\") "
Sep 18 13:50:16.421180 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:16.421109    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd8r6\" (UniqueName: \"kubernetes.io/projected/c07c3d25-46c8-4d16-a94e-1b56c8c4faa9-kube-api-access-sd8r6\") pod \"c07c3d25-46c8-4d16-a94e-1b56c8c4faa9\" (UID: \"c07c3d25-46c8-4d16-a94e-1b56c8c4faa9\") "
Sep 18 13:50:16.421180 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:16.421146    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlhg8\" (UniqueName: \"kubernetes.io/projected/01fef123-0cf2-481a-92c0-67d27d09d812-kube-api-access-dlhg8\") pod \"01fef123-0cf2-481a-92c0-67d27d09d812\" (UID: \"01fef123-0cf2-481a-92c0-67d27d09d812\") "
Sep 18 13:50:16.421440 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:16.421413    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01fef123-0cf2-481a-92c0-67d27d09d812-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "01fef123-0cf2-481a-92c0-67d27d09d812" (UID: "01fef123-0cf2-481a-92c0-67d27d09d812"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:50:16.421790 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:16.421762    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c07c3d25-46c8-4d16-a94e-1b56c8c4faa9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c07c3d25-46c8-4d16-a94e-1b56c8c4faa9" (UID: "c07c3d25-46c8-4d16-a94e-1b56c8c4faa9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:50:16.423551 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:16.423522    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07c3d25-46c8-4d16-a94e-1b56c8c4faa9-kube-api-access-sd8r6" (OuterVolumeSpecName: "kube-api-access-sd8r6") pod "c07c3d25-46c8-4d16-a94e-1b56c8c4faa9" (UID: "c07c3d25-46c8-4d16-a94e-1b56c8c4faa9"). InnerVolumeSpecName "kube-api-access-sd8r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:50:16.423652 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:16.423569    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01fef123-0cf2-481a-92c0-67d27d09d812-kube-api-access-dlhg8" (OuterVolumeSpecName: "kube-api-access-dlhg8") pod "01fef123-0cf2-481a-92c0-67d27d09d812" (UID: "01fef123-0cf2-481a-92c0-67d27d09d812"). InnerVolumeSpecName "kube-api-access-dlhg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:50:16.522361 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:16.522334    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01fef123-0cf2-481a-92c0-67d27d09d812-plugins-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:50:16.522361 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:16.522361    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c07c3d25-46c8-4d16-a94e-1b56c8c4faa9-plugins-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:50:16.522520 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:16.522370    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd8r6\" (UniqueName: \"kubernetes.io/projected/c07c3d25-46c8-4d16-a94e-1b56c8c4faa9-kube-api-access-sd8r6\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:50:16.522520 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:16.522379    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlhg8\" (UniqueName: \"kubernetes.io/projected/01fef123-0cf2-481a-92c0-67d27d09d812-kube-api-access-dlhg8\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:50:17.331771 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:17.331736    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6"
Sep 18 13:50:17.331771 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:17.331759    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6" event={"ID":"c07c3d25-46c8-4d16-a94e-1b56c8c4faa9","Type":"ContainerDied","Data":"70bdc88a6c60e47360b436874d2645e044efb4d856c195bd16b4a424d11b16bb"}
Sep 18 13:50:17.332338 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:17.331815    2586 scope.go:117] "RemoveContainer" containerID="bbde2c73880a3dd5848bbdebee9f192487bb19b8e17d11e72e2056bcbf57b44c"
Sep 18 13:50:17.333455 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:17.333414    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p" event={"ID":"01fef123-0cf2-481a-92c0-67d27d09d812","Type":"ContainerDied","Data":"e313a1e04f044c066abac6a5e56e5a2eb88e73e2dd6bb57822ad624352c5e247"}
Sep 18 13:50:17.333455 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:17.333435    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p"
Sep 18 13:50:17.340871 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:17.340852    2586 scope.go:117] "RemoveContainer" containerID="fa3907f3fdf8b8633148a9a603e73dfb7557f0a4e1c92fa805f5cfe3bfba6478"
Sep 18 13:50:17.353983 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:17.353961    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6"]
Sep 18 13:50:17.360533 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:17.360510    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-eaff22-79bcb8d467-6b8d6"]
Sep 18 13:50:17.371110 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:17.371091    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p"]
Sep 18 13:50:17.375422 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:17.375402    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eaff22-58c5777c8-6l84p"]
Sep 18 13:50:18.850101 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:18.850059    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01fef123-0cf2-481a-92c0-67d27d09d812" path="/var/lib/kubelet/pods/01fef123-0cf2-481a-92c0-67d27d09d812/volumes"
Sep 18 13:50:18.850614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:18.850590    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c07c3d25-46c8-4d16-a94e-1b56c8c4faa9" path="/var/lib/kubelet/pods/c07c3d25-46c8-4d16-a94e-1b56c8c4faa9/volumes"
Sep 18 13:50:20.469648 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:20.469612    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:50:27.563958 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:27.563899    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dd8de-s4mk5"]
Sep 18 13:50:27.564396 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:27.564272    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01fef123-0cf2-481a-92c0-67d27d09d812" containerName="payload-pre-processing"
Sep 18 13:50:27.564396 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:27.564282    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="01fef123-0cf2-481a-92c0-67d27d09d812" containerName="payload-pre-processing"
Sep 18 13:50:27.564396 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:27.564300    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c07c3d25-46c8-4d16-a94e-1b56c8c4faa9" containerName="payload-processing"
Sep 18 13:50:27.564396 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:27.564305    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="c07c3d25-46c8-4d16-a94e-1b56c8c4faa9" containerName="payload-processing"
Sep 18 13:50:27.564396 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:27.564374    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="01fef123-0cf2-481a-92c0-67d27d09d812" containerName="payload-pre-processing"
Sep 18 13:50:27.564396 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:27.564386    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="c07c3d25-46c8-4d16-a94e-1b56c8c4faa9" containerName="payload-processing"
Sep 18 13:50:27.567348 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:27.567330    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dd8de-s4mk5"
Sep 18 13:50:27.569991 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:27.569966    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:50:27.570129 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:27.570101    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nh2cd\""
Sep 18 13:50:27.597438 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:27.597413    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dd8de-s4mk5"]
Sep 18 13:50:27.731242 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:27.731204    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86253aba-c41c-4613-82ba-faa913e52599-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2dd8de-s4mk5\" (UID: \"86253aba-c41c-4613-82ba-faa913e52599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dd8de-s4mk5"
Sep 18 13:50:27.832164 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:27.832084    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86253aba-c41c-4613-82ba-faa913e52599-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2dd8de-s4mk5\" (UID: \"86253aba-c41c-4613-82ba-faa913e52599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dd8de-s4mk5"
Sep 18 13:50:27.832649 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:27.832630    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86253aba-c41c-4613-82ba-faa913e52599-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-2dd8de-s4mk5\" (UID: \"86253aba-c41c-4613-82ba-faa913e52599\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dd8de-s4mk5"
Sep 18 13:50:27.877451 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:27.877428    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dd8de-s4mk5"
Sep 18 13:50:28.162080 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:28.162056    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dd8de-s4mk5"]
Sep 18 13:50:28.163721 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:50:28.163693    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86253aba_c41c_4613_82ba_faa913e52599.slice/crio-fe7b797e954e8eb1bbef9e2601da05446ac4853f99c3e4d97cfb92cefbddf0dd WatchSource:0}: Error finding container fe7b797e954e8eb1bbef9e2601da05446ac4853f99c3e4d97cfb92cefbddf0dd: Status 404 returned error can't find the container with id fe7b797e954e8eb1bbef9e2601da05446ac4853f99c3e4d97cfb92cefbddf0dd
Sep 18 13:50:28.397446 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:28.397416    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dd8de-s4mk5" event={"ID":"86253aba-c41c-4613-82ba-faa913e52599","Type":"ContainerStarted","Data":"fe7b797e954e8eb1bbef9e2601da05446ac4853f99c3e4d97cfb92cefbddf0dd"}
Sep 18 13:50:29.403695 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:29.403663    2586 generic.go:356] "Generic (PLEG): container finished" podID="86253aba-c41c-4613-82ba-faa913e52599" containerID="32785454ee30053b82575855381a1978306b8387bb17ead253bf0a6bc03cb702" exitCode=0
Sep 18 13:50:29.404129 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:29.403720    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dd8de-s4mk5" event={"ID":"86253aba-c41c-4613-82ba-faa913e52599","Type":"ContainerDied","Data":"32785454ee30053b82575855381a1978306b8387bb17ead253bf0a6bc03cb702"}
Sep 18 13:50:30.546380 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:30.546355    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dd8de-s4mk5"
Sep 18 13:50:30.658582 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:30.658544    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86253aba-c41c-4613-82ba-faa913e52599-maas-api-service-ca\") pod \"86253aba-c41c-4613-82ba-faa913e52599\" (UID: \"86253aba-c41c-4613-82ba-faa913e52599\") "
Sep 18 13:50:30.658982 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:30.658958    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86253aba-c41c-4613-82ba-faa913e52599-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "86253aba-c41c-4613-82ba-faa913e52599" (UID: "86253aba-c41c-4613-82ba-faa913e52599"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:50:30.760012 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:30.759938    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86253aba-c41c-4613-82ba-faa913e52599-maas-api-service-ca\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:50:31.415506 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:31.415475    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dd8de-s4mk5"
Sep 18 13:50:31.415693 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:31.415474    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dd8de-s4mk5" event={"ID":"86253aba-c41c-4613-82ba-faa913e52599","Type":"ContainerDied","Data":"fe7b797e954e8eb1bbef9e2601da05446ac4853f99c3e4d97cfb92cefbddf0dd"}
Sep 18 13:50:31.415693 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:31.415627    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe7b797e954e8eb1bbef9e2601da05446ac4853f99c3e4d97cfb92cefbddf0dd"
Sep 18 13:50:40.994140 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:40.994105    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:50:43.422724 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:43.422688    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q"]
Sep 18 13:50:43.423223 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:43.422964    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q" podUID="2b1c489e-2a11-4289-b9fe-928cbf2e98ee" containerName="payload-pre-processing" containerID="cri-o://7265af0bb9332c5a793dcaa26cefd492d84f63a733d2e667ae2f84efa00f3ee8" gracePeriod=30
Sep 18 13:50:43.633373 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:43.633333    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dd8de-s4mk5"]
Sep 18 13:50:43.636723 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:43.636695    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-2dd8de-s4mk5"]
Sep 18 13:50:43.679782 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:43.679726    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q"
Sep 18 13:50:43.778026 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:43.777992    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-js2nj\" (UniqueName: \"kubernetes.io/projected/2b1c489e-2a11-4289-b9fe-928cbf2e98ee-kube-api-access-js2nj\") pod \"2b1c489e-2a11-4289-b9fe-928cbf2e98ee\" (UID: \"2b1c489e-2a11-4289-b9fe-928cbf2e98ee\") "
Sep 18 13:50:43.778219 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:43.778040    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b1c489e-2a11-4289-b9fe-928cbf2e98ee-plugins-config-volume\") pod \"2b1c489e-2a11-4289-b9fe-928cbf2e98ee\" (UID: \"2b1c489e-2a11-4289-b9fe-928cbf2e98ee\") "
Sep 18 13:50:43.778484 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:43.778450    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b1c489e-2a11-4289-b9fe-928cbf2e98ee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2b1c489e-2a11-4289-b9fe-928cbf2e98ee" (UID: "2b1c489e-2a11-4289-b9fe-928cbf2e98ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:50:43.780211 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:43.780187    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1c489e-2a11-4289-b9fe-928cbf2e98ee-kube-api-access-js2nj" (OuterVolumeSpecName: "kube-api-access-js2nj") pod "2b1c489e-2a11-4289-b9fe-928cbf2e98ee" (UID: "2b1c489e-2a11-4289-b9fe-928cbf2e98ee"). InnerVolumeSpecName "kube-api-access-js2nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:50:43.879492 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:43.879448    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js2nj\" (UniqueName: \"kubernetes.io/projected/2b1c489e-2a11-4289-b9fe-928cbf2e98ee-kube-api-access-js2nj\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:50:43.879492 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:43.879481    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b1c489e-2a11-4289-b9fe-928cbf2e98ee-plugins-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:50:44.481897 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:44.481860    2586 generic.go:356] "Generic (PLEG): container finished" podID="2b1c489e-2a11-4289-b9fe-928cbf2e98ee" containerID="7265af0bb9332c5a793dcaa26cefd492d84f63a733d2e667ae2f84efa00f3ee8" exitCode=0
Sep 18 13:50:44.482326 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:44.481934    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q"
Sep 18 13:50:44.482326 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:44.481944    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q" event={"ID":"2b1c489e-2a11-4289-b9fe-928cbf2e98ee","Type":"ContainerDied","Data":"7265af0bb9332c5a793dcaa26cefd492d84f63a733d2e667ae2f84efa00f3ee8"}
Sep 18 13:50:44.482326 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:44.481997    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q" event={"ID":"2b1c489e-2a11-4289-b9fe-928cbf2e98ee","Type":"ContainerDied","Data":"d37a9bc3a05717da7cb29261193db89b050d621a24923fc7f962edf7ca7400bb"}
Sep 18 13:50:44.482326 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:44.482022    2586 scope.go:117] "RemoveContainer" containerID="7265af0bb9332c5a793dcaa26cefd492d84f63a733d2e667ae2f84efa00f3ee8"
Sep 18 13:50:44.491233 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:44.491215    2586 scope.go:117] "RemoveContainer" containerID="7265af0bb9332c5a793dcaa26cefd492d84f63a733d2e667ae2f84efa00f3ee8"
Sep 18 13:50:44.491524 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:50:44.491500    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7265af0bb9332c5a793dcaa26cefd492d84f63a733d2e667ae2f84efa00f3ee8\": container with ID starting with 7265af0bb9332c5a793dcaa26cefd492d84f63a733d2e667ae2f84efa00f3ee8 not found: ID does not exist" containerID="7265af0bb9332c5a793dcaa26cefd492d84f63a733d2e667ae2f84efa00f3ee8"
Sep 18 13:50:44.491588 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:44.491535    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7265af0bb9332c5a793dcaa26cefd492d84f63a733d2e667ae2f84efa00f3ee8"} err="failed to get container status \"7265af0bb9332c5a793dcaa26cefd492d84f63a733d2e667ae2f84efa00f3ee8\": rpc error: code = NotFound desc = could not find container \"7265af0bb9332c5a793dcaa26cefd492d84f63a733d2e667ae2f84efa00f3ee8\": container with ID starting with 7265af0bb9332c5a793dcaa26cefd492d84f63a733d2e667ae2f84efa00f3ee8 not found: ID does not exist"
Sep 18 13:50:44.511958 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:44.511934    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q"]
Sep 18 13:50:44.513668 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:44.513646    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2dd8de-677dfbdbbb-k5z2q"]
Sep 18 13:50:44.849508 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:44.849404    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b1c489e-2a11-4289-b9fe-928cbf2e98ee" path="/var/lib/kubelet/pods/2b1c489e-2a11-4289-b9fe-928cbf2e98ee/volumes"
Sep 18 13:50:44.849768 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:44.849754    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86253aba-c41c-4613-82ba-faa913e52599" path="/var/lib/kubelet/pods/86253aba-c41c-4613-82ba-faa913e52599/volumes"
Sep 18 13:50:49.017888 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:49.017851    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"]
Sep 18 13:50:54.280835 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:50:54.280798    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:51:04.037794 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.037754    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj" podUID="c2dbfe68-4397-40b0-bee5-fc4912111813" containerName="main" containerID="cri-o://edbe80ed86a04c5d7b9a4dbf7156afd8573e03587b99d603287b5e74aaf285ef" gracePeriod=45
Sep 18 13:51:04.281630 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.281600    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:51:04.378170 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.378091    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2dbfe68-4397-40b0-bee5-fc4912111813-home\") pod \"c2dbfe68-4397-40b0-bee5-fc4912111813\" (UID: \"c2dbfe68-4397-40b0-bee5-fc4912111813\") "
Sep 18 13:51:04.378170 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.378143    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s829h\" (UniqueName: \"kubernetes.io/projected/c2dbfe68-4397-40b0-bee5-fc4912111813-kube-api-access-s829h\") pod \"c2dbfe68-4397-40b0-bee5-fc4912111813\" (UID: \"c2dbfe68-4397-40b0-bee5-fc4912111813\") "
Sep 18 13:51:04.378413 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.378253    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2dbfe68-4397-40b0-bee5-fc4912111813-tmp-dir\") pod \"c2dbfe68-4397-40b0-bee5-fc4912111813\" (UID: \"c2dbfe68-4397-40b0-bee5-fc4912111813\") "
Sep 18 13:51:04.378413 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.378303    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2dbfe68-4397-40b0-bee5-fc4912111813-tls-certs\") pod \"c2dbfe68-4397-40b0-bee5-fc4912111813\" (UID: \"c2dbfe68-4397-40b0-bee5-fc4912111813\") "
Sep 18 13:51:04.378413 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.378348    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2dbfe68-4397-40b0-bee5-fc4912111813-home" (OuterVolumeSpecName: "home") pod "c2dbfe68-4397-40b0-bee5-fc4912111813" (UID: "c2dbfe68-4397-40b0-bee5-fc4912111813"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:51:04.378413 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.378354    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2dbfe68-4397-40b0-bee5-fc4912111813-model-cache\") pod \"c2dbfe68-4397-40b0-bee5-fc4912111813\" (UID: \"c2dbfe68-4397-40b0-bee5-fc4912111813\") "
Sep 18 13:51:04.378631 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.378416    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2dbfe68-4397-40b0-bee5-fc4912111813-dshm\") pod \"c2dbfe68-4397-40b0-bee5-fc4912111813\" (UID: \"c2dbfe68-4397-40b0-bee5-fc4912111813\") "
Sep 18 13:51:04.378631 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.378560    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2dbfe68-4397-40b0-bee5-fc4912111813-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c2dbfe68-4397-40b0-bee5-fc4912111813" (UID: "c2dbfe68-4397-40b0-bee5-fc4912111813"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:51:04.378631 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.378585    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2dbfe68-4397-40b0-bee5-fc4912111813-model-cache" (OuterVolumeSpecName: "model-cache") pod "c2dbfe68-4397-40b0-bee5-fc4912111813" (UID: "c2dbfe68-4397-40b0-bee5-fc4912111813"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:51:04.379020 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.378995    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2dbfe68-4397-40b0-bee5-fc4912111813-tmp-dir\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:51:04.379150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.379023    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2dbfe68-4397-40b0-bee5-fc4912111813-model-cache\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:51:04.379150 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.379037    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2dbfe68-4397-40b0-bee5-fc4912111813-home\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:51:04.380457 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.380434    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2dbfe68-4397-40b0-bee5-fc4912111813-kube-api-access-s829h" (OuterVolumeSpecName: "kube-api-access-s829h") pod "c2dbfe68-4397-40b0-bee5-fc4912111813" (UID: "c2dbfe68-4397-40b0-bee5-fc4912111813"). InnerVolumeSpecName "kube-api-access-s829h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:51:04.381018 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.380999    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2dbfe68-4397-40b0-bee5-fc4912111813-dshm" (OuterVolumeSpecName: "dshm") pod "c2dbfe68-4397-40b0-bee5-fc4912111813" (UID: "c2dbfe68-4397-40b0-bee5-fc4912111813"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:51:04.381123 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.381092    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2dbfe68-4397-40b0-bee5-fc4912111813-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c2dbfe68-4397-40b0-bee5-fc4912111813" (UID: "c2dbfe68-4397-40b0-bee5-fc4912111813"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:51:04.480140 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.480107    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s829h\" (UniqueName: \"kubernetes.io/projected/c2dbfe68-4397-40b0-bee5-fc4912111813-kube-api-access-s829h\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:51:04.480140 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.480143    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2dbfe68-4397-40b0-bee5-fc4912111813-tls-certs\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:51:04.480309 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.480153    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2dbfe68-4397-40b0-bee5-fc4912111813-dshm\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:51:04.591289 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.591254    2586 generic.go:356] "Generic (PLEG): container finished" podID="c2dbfe68-4397-40b0-bee5-fc4912111813" containerID="edbe80ed86a04c5d7b9a4dbf7156afd8573e03587b99d603287b5e74aaf285ef" exitCode=0
Sep 18 13:51:04.591434 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.591299    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"
Sep 18 13:51:04.591434 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.591325    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj" event={"ID":"c2dbfe68-4397-40b0-bee5-fc4912111813","Type":"ContainerDied","Data":"edbe80ed86a04c5d7b9a4dbf7156afd8573e03587b99d603287b5e74aaf285ef"}
Sep 18 13:51:04.591434 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.591362    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj" event={"ID":"c2dbfe68-4397-40b0-bee5-fc4912111813","Type":"ContainerDied","Data":"8809e62c7ca3ebf122eb592c694486646e194471cdfb6a6a9bc6aecb18fbf2f4"}
Sep 18 13:51:04.591434 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.591386    2586 scope.go:117] "RemoveContainer" containerID="edbe80ed86a04c5d7b9a4dbf7156afd8573e03587b99d603287b5e74aaf285ef"
Sep 18 13:51:04.600642 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.600623    2586 scope.go:117] "RemoveContainer" containerID="edbe80ed86a04c5d7b9a4dbf7156afd8573e03587b99d603287b5e74aaf285ef"
Sep 18 13:51:04.600921 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:51:04.600877    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edbe80ed86a04c5d7b9a4dbf7156afd8573e03587b99d603287b5e74aaf285ef\": container with ID starting with edbe80ed86a04c5d7b9a4dbf7156afd8573e03587b99d603287b5e74aaf285ef not found: ID does not exist" containerID="edbe80ed86a04c5d7b9a4dbf7156afd8573e03587b99d603287b5e74aaf285ef"
Sep 18 13:51:04.601019 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.600932    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edbe80ed86a04c5d7b9a4dbf7156afd8573e03587b99d603287b5e74aaf285ef"} err="failed to get container status \"edbe80ed86a04c5d7b9a4dbf7156afd8573e03587b99d603287b5e74aaf285ef\": rpc error: code = NotFound desc = could not find container \"edbe80ed86a04c5d7b9a4dbf7156afd8573e03587b99d603287b5e74aaf285ef\": container with ID starting with edbe80ed86a04c5d7b9a4dbf7156afd8573e03587b99d603287b5e74aaf285ef not found: ID does not exist"
Sep 18 13:51:04.616063 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.616037    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"]
Sep 18 13:51:04.624140 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.624117    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-2dd8de/ipp-route-2dd8de-kserve-5d5ddcf9c9-d2plj"]
Sep 18 13:51:04.850442 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:04.850411    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2dbfe68-4397-40b0-bee5-fc4912111813" path="/var/lib/kubelet/pods/c2dbfe68-4397-40b0-bee5-fc4912111813/volumes"
Sep 18 13:51:05.219521 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:05.219489    2586 scope.go:117] "RemoveContainer" containerID="baacfe80e4c697a8ee7ad30866edd86512476d68e0b428ce237c9a95ef4b804c"
Sep 18 13:51:05.227627 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:05.227609    2586 scope.go:117] "RemoveContainer" containerID="53c4670715f697dae3b772bb35d737f12dc9a0c677cb04df80f21ddb182e90e8"
Sep 18 13:51:23.922880 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:23.922842    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"]
Sep 18 13:51:23.923727 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:23.923689    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t" podUID="d3cb21a3-3d69-4284-a76d-fa0a5a35d064" containerName="istio-proxy" containerID="cri-o://997ee940e13779cea5dedc256db7a16ebb7a9ec0f9948d771aaa973877a6ca42" gracePeriod=30
Sep 18 13:51:24.333008 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:24.332924    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f629bd-lw8b8"]
Sep 18 13:51:24.333300 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:24.333288    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86253aba-c41c-4613-82ba-faa913e52599" containerName="revoke-keys"
Sep 18 13:51:24.333347 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:24.333302    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="86253aba-c41c-4613-82ba-faa913e52599" containerName="revoke-keys"
Sep 18 13:51:24.333347 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:24.333313    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b1c489e-2a11-4289-b9fe-928cbf2e98ee" containerName="payload-pre-processing"
Sep 18 13:51:24.333347 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:24.333319    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b1c489e-2a11-4289-b9fe-928cbf2e98ee" containerName="payload-pre-processing"
Sep 18 13:51:24.333347 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:24.333328    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2dbfe68-4397-40b0-bee5-fc4912111813" containerName="main"
Sep 18 13:51:24.333347 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:24.333333    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2dbfe68-4397-40b0-bee5-fc4912111813" containerName="main"
Sep 18 13:51:24.333520 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:24.333386    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="86253aba-c41c-4613-82ba-faa913e52599" containerName="revoke-keys"
Sep 18 13:51:24.333520 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:24.333394    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2dbfe68-4397-40b0-bee5-fc4912111813" containerName="main"
Sep 18 13:51:24.333520 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:24.333401    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b1c489e-2a11-4289-b9fe-928cbf2e98ee" containerName="payload-pre-processing"
Sep 18 13:51:24.337938 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:24.337917    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f629bd-lw8b8"
Sep 18 13:51:24.339819 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:24.339799    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:51:24.339953 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:24.339828    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nh2cd\""
Sep 18 13:51:24.344097 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:24.344071    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f629bd-lw8b8"]
Sep 18 13:51:24.359581 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:24.359557    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3fea84-4c32-4c9f-a3aa-f8ab4374076a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f629bd-lw8b8\" (UID: \"6d3fea84-4c32-4c9f-a3aa-f8ab4374076a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f629bd-lw8b8"
Sep 18 13:51:24.460216 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:24.460181    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3fea84-4c32-4c9f-a3aa-f8ab4374076a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f629bd-lw8b8\" (UID: \"6d3fea84-4c32-4c9f-a3aa-f8ab4374076a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f629bd-lw8b8"
Sep 18 13:51:24.460813 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:24.460788    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3fea84-4c32-4c9f-a3aa-f8ab4374076a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f629bd-lw8b8\" (UID: \"6d3fea84-4c32-4c9f-a3aa-f8ab4374076a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f629bd-lw8b8"
Sep 18 13:51:24.649665 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:24.649637    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f629bd-lw8b8"
Sep 18 13:51:24.934780 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:24.934699    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f629bd-lw8b8"]
Sep 18 13:51:24.938635 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:51:24.938606    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3fea84_4c32_4c9f_a3aa_f8ab4374076a.slice/crio-064ce2744ce8ef6cd72b66de0bf529870cc7e80dda489f19b0fb0dfbef952b44 WatchSource:0}: Error finding container 064ce2744ce8ef6cd72b66de0bf529870cc7e80dda489f19b0fb0dfbef952b44: Status 404 returned error can't find the container with id 064ce2744ce8ef6cd72b66de0bf529870cc7e80dda489f19b0fb0dfbef952b44
Sep 18 13:51:25.694041 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:25.694009    2586 generic.go:356] "Generic (PLEG): container finished" podID="6d3fea84-4c32-4c9f-a3aa-f8ab4374076a" containerID="301a95a191478b7c6e4ca4bf6e2a8874e25366a6b38cf661fcccc1f092cdfbb0" exitCode=0
Sep 18 13:51:25.694216 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:25.694062    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f629bd-lw8b8" event={"ID":"6d3fea84-4c32-4c9f-a3aa-f8ab4374076a","Type":"ContainerDied","Data":"301a95a191478b7c6e4ca4bf6e2a8874e25366a6b38cf661fcccc1f092cdfbb0"}
Sep 18 13:51:25.694216 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:25.694102    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f629bd-lw8b8" event={"ID":"6d3fea84-4c32-4c9f-a3aa-f8ab4374076a","Type":"ContainerStarted","Data":"064ce2744ce8ef6cd72b66de0bf529870cc7e80dda489f19b0fb0dfbef952b44"}
Sep 18 13:51:26.831945 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:26.831920    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f629bd-lw8b8"
Sep 18 13:51:26.883351 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:26.883323    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3fea84-4c32-4c9f-a3aa-f8ab4374076a-maas-api-service-ca\") pod \"6d3fea84-4c32-4c9f-a3aa-f8ab4374076a\" (UID: \"6d3fea84-4c32-4c9f-a3aa-f8ab4374076a\") "
Sep 18 13:51:26.883664 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:26.883639    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d3fea84-4c32-4c9f-a3aa-f8ab4374076a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d3fea84-4c32-4c9f-a3aa-f8ab4374076a" (UID: "6d3fea84-4c32-4c9f-a3aa-f8ab4374076a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:51:26.984283 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:26.984220    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d3fea84-4c32-4c9f-a3aa-f8ab4374076a-maas-api-service-ca\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:51:27.707169 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:27.707137    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f629bd-lw8b8" event={"ID":"6d3fea84-4c32-4c9f-a3aa-f8ab4374076a","Type":"ContainerDied","Data":"064ce2744ce8ef6cd72b66de0bf529870cc7e80dda489f19b0fb0dfbef952b44"}
Sep 18 13:51:27.707169 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:27.707167    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f629bd-lw8b8"
Sep 18 13:51:27.707378 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:27.707182    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="064ce2744ce8ef6cd72b66de0bf529870cc7e80dda489f19b0fb0dfbef952b44"
Sep 18 13:51:29.167168 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.167142    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:51:29.205196 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.205130    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istio-data\") pod \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") "
Sep 18 13:51:29.205196 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.205165    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istio-token\") pod \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") "
Sep 18 13:51:29.205371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.205213    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-workload-socket\") pod \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") "
Sep 18 13:51:29.205371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.205240    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istio-podinfo\") pod \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") "
Sep 18 13:51:29.205371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.205333    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istio-envoy\") pod \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") "
Sep 18 13:51:29.205371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.205358    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istiod-ca-cert\") pod \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") "
Sep 18 13:51:29.205583 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.205390    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-workload-certs\") pod \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") "
Sep 18 13:51:29.205583 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.205421    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qq5c\" (UniqueName: \"kubernetes.io/projected/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-kube-api-access-5qq5c\") pod \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") "
Sep 18 13:51:29.205583 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.205457    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-credential-socket\") pod \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\" (UID: \"d3cb21a3-3d69-4284-a76d-fa0a5a35d064\") "
Sep 18 13:51:29.205583 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.205480    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istio-data" (OuterVolumeSpecName: "istio-data") pod "d3cb21a3-3d69-4284-a76d-fa0a5a35d064" (UID: "d3cb21a3-3d69-4284-a76d-fa0a5a35d064"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:51:29.205583 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.205512    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d3cb21a3-3d69-4284-a76d-fa0a5a35d064" (UID: "d3cb21a3-3d69-4284-a76d-fa0a5a35d064"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:51:29.205844 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.205717    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d3cb21a3-3d69-4284-a76d-fa0a5a35d064" (UID: "d3cb21a3-3d69-4284-a76d-fa0a5a35d064"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:51:29.205844 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.205749    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d3cb21a3-3d69-4284-a76d-fa0a5a35d064" (UID: "d3cb21a3-3d69-4284-a76d-fa0a5a35d064"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:51:29.205844 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.205810    2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-workload-socket\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:51:29.205844 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.205828    2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-workload-certs\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:51:29.205844 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.205842    2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istio-data\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:51:29.206300 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.205924    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d3cb21a3-3d69-4284-a76d-fa0a5a35d064" (UID: "d3cb21a3-3d69-4284-a76d-fa0a5a35d064"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:51:29.207722 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.207690    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d3cb21a3-3d69-4284-a76d-fa0a5a35d064" (UID: "d3cb21a3-3d69-4284-a76d-fa0a5a35d064"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:51:29.207829 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.207787    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d3cb21a3-3d69-4284-a76d-fa0a5a35d064" (UID: "d3cb21a3-3d69-4284-a76d-fa0a5a35d064"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 13:51:29.208265 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.208240    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istio-token" (OuterVolumeSpecName: "istio-token") pod "d3cb21a3-3d69-4284-a76d-fa0a5a35d064" (UID: "d3cb21a3-3d69-4284-a76d-fa0a5a35d064"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:51:29.208331 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.208248    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-kube-api-access-5qq5c" (OuterVolumeSpecName: "kube-api-access-5qq5c") pod "d3cb21a3-3d69-4284-a76d-fa0a5a35d064" (UID: "d3cb21a3-3d69-4284-a76d-fa0a5a35d064"). InnerVolumeSpecName "kube-api-access-5qq5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:51:29.307110 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.307069    2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istio-envoy\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:51:29.307110 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.307110    2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istiod-ca-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:51:29.307267 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.307123    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qq5c\" (UniqueName: \"kubernetes.io/projected/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-kube-api-access-5qq5c\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:51:29.307267 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.307136    2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-credential-socket\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:51:29.307267 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.307148    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istio-token\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:51:29.307267 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.307159    2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3cb21a3-3d69-4284-a76d-fa0a5a35d064-istio-podinfo\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:51:29.719241 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.719202    2586 generic.go:356] "Generic (PLEG): container finished" podID="d3cb21a3-3d69-4284-a76d-fa0a5a35d064" containerID="997ee940e13779cea5dedc256db7a16ebb7a9ec0f9948d771aaa973877a6ca42" exitCode=0
Sep 18 13:51:29.719390 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.719258    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"
Sep 18 13:51:29.719390 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.719285    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t" event={"ID":"d3cb21a3-3d69-4284-a76d-fa0a5a35d064","Type":"ContainerDied","Data":"997ee940e13779cea5dedc256db7a16ebb7a9ec0f9948d771aaa973877a6ca42"}
Sep 18 13:51:29.719390 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.719341    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t" event={"ID":"d3cb21a3-3d69-4284-a76d-fa0a5a35d064","Type":"ContainerDied","Data":"66a0d4157871300f2e24503b368e8a6c2ebe1df50e3562af142f7c8a1102718d"}
Sep 18 13:51:29.719390 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.719363    2586 scope.go:117] "RemoveContainer" containerID="997ee940e13779cea5dedc256db7a16ebb7a9ec0f9948d771aaa973877a6ca42"
Sep 18 13:51:29.728698 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.728679    2586 scope.go:117] "RemoveContainer" containerID="a92e06f4dfc92da613b6a2e2a6a7718f8100fa3e8fe39b280f4c8944b653589e"
Sep 18 13:51:29.736535 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.736515    2586 scope.go:117] "RemoveContainer" containerID="997ee940e13779cea5dedc256db7a16ebb7a9ec0f9948d771aaa973877a6ca42"
Sep 18 13:51:29.736793 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:51:29.736774    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"997ee940e13779cea5dedc256db7a16ebb7a9ec0f9948d771aaa973877a6ca42\": container with ID starting with 997ee940e13779cea5dedc256db7a16ebb7a9ec0f9948d771aaa973877a6ca42 not found: ID does not exist" containerID="997ee940e13779cea5dedc256db7a16ebb7a9ec0f9948d771aaa973877a6ca42"
Sep 18 13:51:29.736853 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.736801    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"997ee940e13779cea5dedc256db7a16ebb7a9ec0f9948d771aaa973877a6ca42"} err="failed to get container status \"997ee940e13779cea5dedc256db7a16ebb7a9ec0f9948d771aaa973877a6ca42\": rpc error: code = NotFound desc = could not find container \"997ee940e13779cea5dedc256db7a16ebb7a9ec0f9948d771aaa973877a6ca42\": container with ID starting with 997ee940e13779cea5dedc256db7a16ebb7a9ec0f9948d771aaa973877a6ca42 not found: ID does not exist"
Sep 18 13:51:29.736853 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.736816    2586 scope.go:117] "RemoveContainer" containerID="a92e06f4dfc92da613b6a2e2a6a7718f8100fa3e8fe39b280f4c8944b653589e"
Sep 18 13:51:29.737072 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:51:29.737057    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a92e06f4dfc92da613b6a2e2a6a7718f8100fa3e8fe39b280f4c8944b653589e\": container with ID starting with a92e06f4dfc92da613b6a2e2a6a7718f8100fa3e8fe39b280f4c8944b653589e not found: ID does not exist" containerID="a92e06f4dfc92da613b6a2e2a6a7718f8100fa3e8fe39b280f4c8944b653589e"
Sep 18 13:51:29.737113 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.737077    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a92e06f4dfc92da613b6a2e2a6a7718f8100fa3e8fe39b280f4c8944b653589e"} err="failed to get container status \"a92e06f4dfc92da613b6a2e2a6a7718f8100fa3e8fe39b280f4c8944b653589e\": rpc error: code = NotFound desc = could not find container \"a92e06f4dfc92da613b6a2e2a6a7718f8100fa3e8fe39b280f4c8944b653589e\": container with ID starting with a92e06f4dfc92da613b6a2e2a6a7718f8100fa3e8fe39b280f4c8944b653589e not found: ID does not exist"
Sep 18 13:51:29.740658 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.740637    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"]
Sep 18 13:51:29.746101 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:29.746081    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2dd8de-openshift-default-698d8489fb-bvc9t"]
Sep 18 13:51:30.849393 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:30.849362    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3cb21a3-3d69-4284-a76d-fa0a5a35d064" path="/var/lib/kubelet/pods/d3cb21a3-3d69-4284-a76d-fa0a5a35d064/volumes"
Sep 18 13:51:35.217812 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:35.217770    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f629bd-lw8b8"]
Sep 18 13:51:35.221213 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:35.221187    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f629bd-lw8b8"]
Sep 18 13:51:36.849547 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:36.849512    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d3fea84-4c32-4c9f-a3aa-f8ab4374076a" path="/var/lib/kubelet/pods/6d3fea84-4c32-4c9f-a3aa-f8ab4374076a/volumes"
Sep 18 13:51:38.585741 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:38.585705    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:51:42.750723 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:42.750688    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"]
Sep 18 13:51:42.751143 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:42.750943    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh" podUID="2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65" containerName="istio-proxy" containerID="cri-o://e9bed24d4edf2f234cd05a52cd5e38934c725f5356dd8ee6024d7b7f6d133d4d" gracePeriod=30
Sep 18 13:51:47.998970 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:47.998943    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:51:48.083545 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.083485    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istio-data\") pod \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") "
Sep 18 13:51:48.083545 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.083526    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-credential-socket\") pod \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") "
Sep 18 13:51:48.083749 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.083565    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccmsg\" (UniqueName: \"kubernetes.io/projected/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-kube-api-access-ccmsg\") pod \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") "
Sep 18 13:51:48.083749 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.083617    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-workload-certs\") pod \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") "
Sep 18 13:51:48.083749 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.083663    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istio-token\") pod \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") "
Sep 18 13:51:48.083749 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.083689    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istio-podinfo\") pod \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") "
Sep 18 13:51:48.083749 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.083714    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istiod-ca-cert\") pod \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") "
Sep 18 13:51:48.083749 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.083749    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istio-envoy\") pod \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") "
Sep 18 13:51:48.084101 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.083768    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istio-data" (OuterVolumeSpecName: "istio-data") pod "2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65" (UID: "2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:51:48.084101 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.083780    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-workload-socket\") pod \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\" (UID: \"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65\") "
Sep 18 13:51:48.084101 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.083869    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65" (UID: "2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:51:48.084101 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.084094    2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istio-data\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:51:48.084303 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.084115    2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-credential-socket\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:51:48.084303 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.084120    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65" (UID: "2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:51:48.084451 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.084427    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65" (UID: "2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:51:48.084516 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.084490    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65" (UID: "2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:51:48.086095 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.086059    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65" (UID: "2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 13:51:48.086205 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.086167    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istio-token" (OuterVolumeSpecName: "istio-token") pod "2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65" (UID: "2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:51:48.086490 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.086466    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-kube-api-access-ccmsg" (OuterVolumeSpecName: "kube-api-access-ccmsg") pod "2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65" (UID: "2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65"). InnerVolumeSpecName "kube-api-access-ccmsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:51:48.086560 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.086460    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65" (UID: "2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:51:48.185106 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.185081    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccmsg\" (UniqueName: \"kubernetes.io/projected/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-kube-api-access-ccmsg\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:51:48.185106 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.185104    2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-workload-certs\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:51:48.185256 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.185114    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istio-token\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:51:48.185256 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.185122    2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istio-podinfo\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:51:48.185256 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.185130    2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istiod-ca-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:51:48.185256 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.185137    2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-istio-envoy\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:51:48.185256 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.185144    2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65-workload-socket\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:51:48.813365 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.813326    2586 generic.go:356] "Generic (PLEG): container finished" podID="2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65" containerID="e9bed24d4edf2f234cd05a52cd5e38934c725f5356dd8ee6024d7b7f6d133d4d" exitCode=0
Sep 18 13:51:48.813562 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.813374    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"
Sep 18 13:51:48.813562 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.813412    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh" event={"ID":"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65","Type":"ContainerDied","Data":"e9bed24d4edf2f234cd05a52cd5e38934c725f5356dd8ee6024d7b7f6d133d4d"}
Sep 18 13:51:48.813562 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.813478    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh" event={"ID":"2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65","Type":"ContainerDied","Data":"77b9991b9256569ade994275ab29586db8b9d46d3f225a5a4ef6fdb0713e772b"}
Sep 18 13:51:48.813562 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.813500    2586 scope.go:117] "RemoveContainer" containerID="e9bed24d4edf2f234cd05a52cd5e38934c725f5356dd8ee6024d7b7f6d133d4d"
Sep 18 13:51:48.822658 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.822638    2586 scope.go:117] "RemoveContainer" containerID="ddb662268bdc55bb5f78278232492f940e86bd275f1098b37c6395d9bedabdaf"
Sep 18 13:51:48.830318 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.830300    2586 scope.go:117] "RemoveContainer" containerID="e9bed24d4edf2f234cd05a52cd5e38934c725f5356dd8ee6024d7b7f6d133d4d"
Sep 18 13:51:48.830589 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:51:48.830571    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9bed24d4edf2f234cd05a52cd5e38934c725f5356dd8ee6024d7b7f6d133d4d\": container with ID starting with e9bed24d4edf2f234cd05a52cd5e38934c725f5356dd8ee6024d7b7f6d133d4d not found: ID does not exist" containerID="e9bed24d4edf2f234cd05a52cd5e38934c725f5356dd8ee6024d7b7f6d133d4d"
Sep 18 13:51:48.830627 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.830599    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9bed24d4edf2f234cd05a52cd5e38934c725f5356dd8ee6024d7b7f6d133d4d"} err="failed to get container status \"e9bed24d4edf2f234cd05a52cd5e38934c725f5356dd8ee6024d7b7f6d133d4d\": rpc error: code = NotFound desc = could not find container \"e9bed24d4edf2f234cd05a52cd5e38934c725f5356dd8ee6024d7b7f6d133d4d\": container with ID starting with e9bed24d4edf2f234cd05a52cd5e38934c725f5356dd8ee6024d7b7f6d133d4d not found: ID does not exist"
Sep 18 13:51:48.830627 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.830616    2586 scope.go:117] "RemoveContainer" containerID="ddb662268bdc55bb5f78278232492f940e86bd275f1098b37c6395d9bedabdaf"
Sep 18 13:51:48.830855 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:51:48.830837    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddb662268bdc55bb5f78278232492f940e86bd275f1098b37c6395d9bedabdaf\": container with ID starting with ddb662268bdc55bb5f78278232492f940e86bd275f1098b37c6395d9bedabdaf not found: ID does not exist" containerID="ddb662268bdc55bb5f78278232492f940e86bd275f1098b37c6395d9bedabdaf"
Sep 18 13:51:48.830892 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.830860    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddb662268bdc55bb5f78278232492f940e86bd275f1098b37c6395d9bedabdaf"} err="failed to get container status \"ddb662268bdc55bb5f78278232492f940e86bd275f1098b37c6395d9bedabdaf\": rpc error: code = NotFound desc = could not find container \"ddb662268bdc55bb5f78278232492f940e86bd275f1098b37c6395d9bedabdaf\": container with ID starting with ddb662268bdc55bb5f78278232492f940e86bd275f1098b37c6395d9bedabdaf not found: ID does not exist"
Sep 18 13:51:48.834142 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.834122    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"]
Sep 18 13:51:48.840587 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.840565    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f629bd-openshift-default-77894b9698-jfkkh"]
Sep 18 13:51:48.849024 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:48.849003    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65" path="/var/lib/kubelet/pods/2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65/volumes"
Sep 18 13:51:53.871134 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:53.871092    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-537abe-29828985-2rcvq"]
Sep 18 13:51:53.874260 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:53.874232    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-537abe-29828985-2rcvq"]
Sep 18 13:51:54.849438 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:54.849396    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b08413ad-899a-4368-bfe3-24c1385afb9a" path="/var/lib/kubelet/pods/b08413ad-899a-4368-bfe3-24c1385afb9a/volumes"
Sep 18 13:51:56.900499 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:51:56.900469    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:52:01.457672 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:01.457631    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"]
Sep 18 13:52:01.458152 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:01.457888    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959" podUID="7e004867-2880-4eda-8770-f1e6fb00eb8d" containerName="istio-proxy" containerID="cri-o://23c560272e5d33419f62287446d9b3c6ce64306f60871d8a57e417c18b8a40cf" gracePeriod=30
Sep 18 13:52:05.262050 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:05.262017    2586 scope.go:117] "RemoveContainer" containerID="22cff72cd8cb97f5a9cf45b94ef685947ea6d96e9aab93fc76c737fbe3a2ea49"
Sep 18 13:52:06.707753 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.707731    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:52:06.758152 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.758077    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smkjc\" (UniqueName: \"kubernetes.io/projected/7e004867-2880-4eda-8770-f1e6fb00eb8d-kube-api-access-smkjc\") pod \"7e004867-2880-4eda-8770-f1e6fb00eb8d\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") "
Sep 18 13:52:06.758152 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.758114    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-istio-data\") pod \"7e004867-2880-4eda-8770-f1e6fb00eb8d\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") "
Sep 18 13:52:06.758152 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.758151    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-credential-socket\") pod \"7e004867-2880-4eda-8770-f1e6fb00eb8d\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") "
Sep 18 13:52:06.758423 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.758174    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-istio-envoy\") pod \"7e004867-2880-4eda-8770-f1e6fb00eb8d\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") "
Sep 18 13:52:06.758423 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.758231    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-workload-socket\") pod \"7e004867-2880-4eda-8770-f1e6fb00eb8d\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") "
Sep 18 13:52:06.758423 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.758278    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e004867-2880-4eda-8770-f1e6fb00eb8d-istio-token\") pod \"7e004867-2880-4eda-8770-f1e6fb00eb8d\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") "
Sep 18 13:52:06.758423 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.758312    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-workload-certs\") pod \"7e004867-2880-4eda-8770-f1e6fb00eb8d\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") "
Sep 18 13:52:06.758423 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.758346    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-istio-data" (OuterVolumeSpecName: "istio-data") pod "7e004867-2880-4eda-8770-f1e6fb00eb8d" (UID: "7e004867-2880-4eda-8770-f1e6fb00eb8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:52:06.758423 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.758365    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e004867-2880-4eda-8770-f1e6fb00eb8d-istiod-ca-cert\") pod \"7e004867-2880-4eda-8770-f1e6fb00eb8d\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") "
Sep 18 13:52:06.758423 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.758411    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e004867-2880-4eda-8770-f1e6fb00eb8d-istio-podinfo\") pod \"7e004867-2880-4eda-8770-f1e6fb00eb8d\" (UID: \"7e004867-2880-4eda-8770-f1e6fb00eb8d\") "
Sep 18 13:52:06.758802 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.758512    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7e004867-2880-4eda-8770-f1e6fb00eb8d" (UID: "7e004867-2880-4eda-8770-f1e6fb00eb8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:52:06.758802 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.758552    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7e004867-2880-4eda-8770-f1e6fb00eb8d" (UID: "7e004867-2880-4eda-8770-f1e6fb00eb8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:52:06.758802 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.758654    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7e004867-2880-4eda-8770-f1e6fb00eb8d" (UID: "7e004867-2880-4eda-8770-f1e6fb00eb8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:52:06.758802 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.758778    2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-workload-socket\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:52:06.758802 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.758798    2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-workload-certs\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:52:06.759115 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.758811    2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-istio-data\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:52:06.759115 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.758803    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e004867-2880-4eda-8770-f1e6fb00eb8d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7e004867-2880-4eda-8770-f1e6fb00eb8d" (UID: "7e004867-2880-4eda-8770-f1e6fb00eb8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:52:06.759115 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.758826    2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-credential-socket\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:52:06.760687 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.760660    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e004867-2880-4eda-8770-f1e6fb00eb8d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7e004867-2880-4eda-8770-f1e6fb00eb8d" (UID: "7e004867-2880-4eda-8770-f1e6fb00eb8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 13:52:06.760785 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.760707    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e004867-2880-4eda-8770-f1e6fb00eb8d-istio-token" (OuterVolumeSpecName: "istio-token") pod "7e004867-2880-4eda-8770-f1e6fb00eb8d" (UID: "7e004867-2880-4eda-8770-f1e6fb00eb8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:52:06.761042 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.761024    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e004867-2880-4eda-8770-f1e6fb00eb8d-kube-api-access-smkjc" (OuterVolumeSpecName: "kube-api-access-smkjc") pod "7e004867-2880-4eda-8770-f1e6fb00eb8d" (UID: "7e004867-2880-4eda-8770-f1e6fb00eb8d"). InnerVolumeSpecName "kube-api-access-smkjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:52:06.761096 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.761073    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7e004867-2880-4eda-8770-f1e6fb00eb8d" (UID: "7e004867-2880-4eda-8770-f1e6fb00eb8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:52:06.860130 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.860111    2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e004867-2880-4eda-8770-f1e6fb00eb8d-istiod-ca-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:52:06.860218 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.860132    2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e004867-2880-4eda-8770-f1e6fb00eb8d-istio-podinfo\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:52:06.860218 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.860141    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smkjc\" (UniqueName: \"kubernetes.io/projected/7e004867-2880-4eda-8770-f1e6fb00eb8d-kube-api-access-smkjc\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:52:06.860218 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.860151    2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e004867-2880-4eda-8770-f1e6fb00eb8d-istio-envoy\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:52:06.860218 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.860159    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e004867-2880-4eda-8770-f1e6fb00eb8d-istio-token\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:52:06.901700 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.901668    2586 generic.go:356] "Generic (PLEG): container finished" podID="7e004867-2880-4eda-8770-f1e6fb00eb8d" containerID="23c560272e5d33419f62287446d9b3c6ce64306f60871d8a57e417c18b8a40cf" exitCode=0
Sep 18 13:52:06.901818 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.901704    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959" event={"ID":"7e004867-2880-4eda-8770-f1e6fb00eb8d","Type":"ContainerDied","Data":"23c560272e5d33419f62287446d9b3c6ce64306f60871d8a57e417c18b8a40cf"}
Sep 18 13:52:06.901818 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.901748    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959" event={"ID":"7e004867-2880-4eda-8770-f1e6fb00eb8d","Type":"ContainerDied","Data":"976306bd77fc4214fa5c69fea48d4d1dfde6440bc26a1dedc125be7f64f4ec6f"}
Sep 18 13:52:06.901818 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.901778    2586 scope.go:117] "RemoveContainer" containerID="23c560272e5d33419f62287446d9b3c6ce64306f60871d8a57e417c18b8a40cf"
Sep 18 13:52:06.901818 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.901711    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"
Sep 18 13:52:06.910712 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.910696    2586 scope.go:117] "RemoveContainer" containerID="8174ba9c274f09f48b0d3a347e5a7989a886efe6c21c86b826d205548ef55119"
Sep 18 13:52:06.919335 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.919313    2586 scope.go:117] "RemoveContainer" containerID="23c560272e5d33419f62287446d9b3c6ce64306f60871d8a57e417c18b8a40cf"
Sep 18 13:52:06.919629 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:52:06.919609    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23c560272e5d33419f62287446d9b3c6ce64306f60871d8a57e417c18b8a40cf\": container with ID starting with 23c560272e5d33419f62287446d9b3c6ce64306f60871d8a57e417c18b8a40cf not found: ID does not exist" containerID="23c560272e5d33419f62287446d9b3c6ce64306f60871d8a57e417c18b8a40cf"
Sep 18 13:52:06.919704 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.919637    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23c560272e5d33419f62287446d9b3c6ce64306f60871d8a57e417c18b8a40cf"} err="failed to get container status \"23c560272e5d33419f62287446d9b3c6ce64306f60871d8a57e417c18b8a40cf\": rpc error: code = NotFound desc = could not find container \"23c560272e5d33419f62287446d9b3c6ce64306f60871d8a57e417c18b8a40cf\": container with ID starting with 23c560272e5d33419f62287446d9b3c6ce64306f60871d8a57e417c18b8a40cf not found: ID does not exist"
Sep 18 13:52:06.919704 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.919653    2586 scope.go:117] "RemoveContainer" containerID="8174ba9c274f09f48b0d3a347e5a7989a886efe6c21c86b826d205548ef55119"
Sep 18 13:52:06.919832 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.919743    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"]
Sep 18 13:52:06.919940 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:52:06.919921    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8174ba9c274f09f48b0d3a347e5a7989a886efe6c21c86b826d205548ef55119\": container with ID starting with 8174ba9c274f09f48b0d3a347e5a7989a886efe6c21c86b826d205548ef55119 not found: ID does not exist" containerID="8174ba9c274f09f48b0d3a347e5a7989a886efe6c21c86b826d205548ef55119"
Sep 18 13:52:06.919992 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.919946    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8174ba9c274f09f48b0d3a347e5a7989a886efe6c21c86b826d205548ef55119"} err="failed to get container status \"8174ba9c274f09f48b0d3a347e5a7989a886efe6c21c86b826d205548ef55119\": rpc error: code = NotFound desc = could not find container \"8174ba9c274f09f48b0d3a347e5a7989a886efe6c21c86b826d205548ef55119\": container with ID starting with 8174ba9c274f09f48b0d3a347e5a7989a886efe6c21c86b826d205548ef55119 not found: ID does not exist"
Sep 18 13:52:06.924874 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:06.924852    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-537abe-openshift-default-5467898dc8-k2959"]
Sep 18 13:52:08.849675 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:08.849639    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e004867-2880-4eda-8770-f1e6fb00eb8d" path="/var/lib/kubelet/pods/7e004867-2880-4eda-8770-f1e6fb00eb8d/volumes"
Sep 18 13:52:12.438240 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:12.438206    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf"]
Sep 18 13:52:12.438654 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:12.438457    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf" podUID="8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b" containerName="payload-pre-processing" containerID="cri-o://ab7636577500e2b0927b41d46a9e044412191fce786b8e48ba0e68fd870c4176" gracePeriod=30
Sep 18 13:52:12.683941 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:12.683914    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf"
Sep 18 13:52:12.807387 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:12.807304    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b-plugins-config-volume\") pod \"8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b\" (UID: \"8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b\") "
Sep 18 13:52:12.807387 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:12.807381    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kl4z6\" (UniqueName: \"kubernetes.io/projected/8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b-kube-api-access-kl4z6\") pod \"8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b\" (UID: \"8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b\") "
Sep 18 13:52:12.807680 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:12.807644    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b" (UID: "8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:52:12.809536 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:12.809509    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b-kube-api-access-kl4z6" (OuterVolumeSpecName: "kube-api-access-kl4z6") pod "8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b" (UID: "8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b"). InnerVolumeSpecName "kube-api-access-kl4z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:52:12.908465 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:12.908422    2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b-plugins-config-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:52:12.908465 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:12.908456    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl4z6\" (UniqueName: \"kubernetes.io/projected/8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b-kube-api-access-kl4z6\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:52:12.931248 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:12.931222    2586 generic.go:356] "Generic (PLEG): container finished" podID="8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b" containerID="ab7636577500e2b0927b41d46a9e044412191fce786b8e48ba0e68fd870c4176" exitCode=0
Sep 18 13:52:12.931398 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:12.931277    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf"
Sep 18 13:52:12.931398 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:12.931286    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf" event={"ID":"8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b","Type":"ContainerDied","Data":"ab7636577500e2b0927b41d46a9e044412191fce786b8e48ba0e68fd870c4176"}
Sep 18 13:52:12.931398 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:12.931328    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf" event={"ID":"8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b","Type":"ContainerDied","Data":"5453eb2d8e818186117818b04259aff44ad0e47d20ac41e7b343af9a5728c273"}
Sep 18 13:52:12.931398 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:12.931349    2586 scope.go:117] "RemoveContainer" containerID="ab7636577500e2b0927b41d46a9e044412191fce786b8e48ba0e68fd870c4176"
Sep 18 13:52:12.939745 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:12.939730    2586 scope.go:117] "RemoveContainer" containerID="ab7636577500e2b0927b41d46a9e044412191fce786b8e48ba0e68fd870c4176"
Sep 18 13:52:12.940046 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:52:12.940023    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab7636577500e2b0927b41d46a9e044412191fce786b8e48ba0e68fd870c4176\": container with ID starting with ab7636577500e2b0927b41d46a9e044412191fce786b8e48ba0e68fd870c4176 not found: ID does not exist" containerID="ab7636577500e2b0927b41d46a9e044412191fce786b8e48ba0e68fd870c4176"
Sep 18 13:52:12.940111 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:12.940053    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7636577500e2b0927b41d46a9e044412191fce786b8e48ba0e68fd870c4176"} err="failed to get container status \"ab7636577500e2b0927b41d46a9e044412191fce786b8e48ba0e68fd870c4176\": rpc error: code = NotFound desc = could not find container \"ab7636577500e2b0927b41d46a9e044412191fce786b8e48ba0e68fd870c4176\": container with ID starting with ab7636577500e2b0927b41d46a9e044412191fce786b8e48ba0e68fd870c4176 not found: ID does not exist"
Sep 18 13:52:12.949142 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:12.949120    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf"]
Sep 18 13:52:12.954317 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:12.954297    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c50e10-566fc47978-d6tgf"]
Sep 18 13:52:14.849531 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:14.849449    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b" path="/var/lib/kubelet/pods/8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b/volumes"
Sep 18 13:52:15.095156 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:15.095119    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:52:20.503877 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:20.503842    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"]
Sep 18 13:52:20.504333 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:20.504103    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7" podUID="5f712e91-6f18-4fe4-9f0b-5550d6a5e368" containerName="istio-proxy" containerID="cri-o://968fe0f23546141ede6d094fb4c361640a2d335b9af196766b69221871be78e3" gracePeriod=30
Sep 18 13:52:24.676914 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:24.676873    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:52:25.739393 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.739364    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:52:25.816479 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.816409    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istio-podinfo\") pod \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") "
Sep 18 13:52:25.816479 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.816446    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istiod-ca-cert\") pod \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") "
Sep 18 13:52:25.816479 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.816479    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-workload-certs\") pod \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") "
Sep 18 13:52:25.816724 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.816506    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istio-data\") pod \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") "
Sep 18 13:52:25.816724 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.816525    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4mzd\" (UniqueName: \"kubernetes.io/projected/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-kube-api-access-x4mzd\") pod \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") "
Sep 18 13:52:25.816724 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.816553    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-credential-socket\") pod \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") "
Sep 18 13:52:25.816724 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.816574    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istio-token\") pod \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") "
Sep 18 13:52:25.816724 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.816598    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istio-envoy\") pod \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") "
Sep 18 13:52:25.816724 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.816639    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-workload-socket\") pod \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\" (UID: \"5f712e91-6f18-4fe4-9f0b-5550d6a5e368\") "
Sep 18 13:52:25.817072 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.816808    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istio-data" (OuterVolumeSpecName: "istio-data") pod "5f712e91-6f18-4fe4-9f0b-5550d6a5e368" (UID: "5f712e91-6f18-4fe4-9f0b-5550d6a5e368"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:52:25.817072 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.816819    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5f712e91-6f18-4fe4-9f0b-5550d6a5e368" (UID: "5f712e91-6f18-4fe4-9f0b-5550d6a5e368"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:52:25.817072 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.816836    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5f712e91-6f18-4fe4-9f0b-5550d6a5e368" (UID: "5f712e91-6f18-4fe4-9f0b-5550d6a5e368"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:52:25.817238 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.817070    2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istiod-ca-cert\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:52:25.817238 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.817089    2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-workload-certs\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:52:25.817238 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.817103    2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istio-data\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:52:25.817238 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.817050    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5f712e91-6f18-4fe4-9f0b-5550d6a5e368" (UID: "5f712e91-6f18-4fe4-9f0b-5550d6a5e368"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:52:25.817238 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.817124    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5f712e91-6f18-4fe4-9f0b-5550d6a5e368" (UID: "5f712e91-6f18-4fe4-9f0b-5550d6a5e368"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:52:25.818727 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.818697    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5f712e91-6f18-4fe4-9f0b-5550d6a5e368" (UID: "5f712e91-6f18-4fe4-9f0b-5550d6a5e368"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 13:52:25.819050 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.819023    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istio-token" (OuterVolumeSpecName: "istio-token") pod "5f712e91-6f18-4fe4-9f0b-5550d6a5e368" (UID: "5f712e91-6f18-4fe4-9f0b-5550d6a5e368"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:52:25.819440 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.819422    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5f712e91-6f18-4fe4-9f0b-5550d6a5e368" (UID: "5f712e91-6f18-4fe4-9f0b-5550d6a5e368"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:52:25.819496 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.819435    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-kube-api-access-x4mzd" (OuterVolumeSpecName: "kube-api-access-x4mzd") pod "5f712e91-6f18-4fe4-9f0b-5550d6a5e368" (UID: "5f712e91-6f18-4fe4-9f0b-5550d6a5e368"). InnerVolumeSpecName "kube-api-access-x4mzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:52:25.917828 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.917798    2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istio-podinfo\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:52:25.917828 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.917826    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4mzd\" (UniqueName: \"kubernetes.io/projected/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-kube-api-access-x4mzd\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:52:25.918009 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.917840    2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-credential-socket\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:52:25.918009 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.917852    2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istio-token\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:52:25.918009 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.917863    2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-istio-envoy\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:52:25.918009 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.917874    2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f712e91-6f18-4fe4-9f0b-5550d6a5e368-workload-socket\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:52:25.991740 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.991713    2586 generic.go:356] "Generic (PLEG): container finished" podID="5f712e91-6f18-4fe4-9f0b-5550d6a5e368" containerID="968fe0f23546141ede6d094fb4c361640a2d335b9af196766b69221871be78e3" exitCode=0
Sep 18 13:52:25.991841 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.991769    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"
Sep 18 13:52:25.991841 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.991799    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7" event={"ID":"5f712e91-6f18-4fe4-9f0b-5550d6a5e368","Type":"ContainerDied","Data":"968fe0f23546141ede6d094fb4c361640a2d335b9af196766b69221871be78e3"}
Sep 18 13:52:25.991938 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.991858    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7" event={"ID":"5f712e91-6f18-4fe4-9f0b-5550d6a5e368","Type":"ContainerDied","Data":"01ef8180234e89f3da22527756733a7719a80723cc5ed259f0a57e4833e74da8"}
Sep 18 13:52:25.991938 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:25.991890    2586 scope.go:117] "RemoveContainer" containerID="968fe0f23546141ede6d094fb4c361640a2d335b9af196766b69221871be78e3"
Sep 18 13:52:26.001355 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:26.001339    2586 scope.go:117] "RemoveContainer" containerID="0f023bb75641e724c5216e597428665c64c6cf4313c023cb8ab820f64fbde425"
Sep 18 13:52:26.008995 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:26.008975    2586 scope.go:117] "RemoveContainer" containerID="968fe0f23546141ede6d094fb4c361640a2d335b9af196766b69221871be78e3"
Sep 18 13:52:26.009282 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:52:26.009262    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"968fe0f23546141ede6d094fb4c361640a2d335b9af196766b69221871be78e3\": container with ID starting with 968fe0f23546141ede6d094fb4c361640a2d335b9af196766b69221871be78e3 not found: ID does not exist" containerID="968fe0f23546141ede6d094fb4c361640a2d335b9af196766b69221871be78e3"
Sep 18 13:52:26.009337 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:26.009290    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"968fe0f23546141ede6d094fb4c361640a2d335b9af196766b69221871be78e3"} err="failed to get container status \"968fe0f23546141ede6d094fb4c361640a2d335b9af196766b69221871be78e3\": rpc error: code = NotFound desc = could not find container \"968fe0f23546141ede6d094fb4c361640a2d335b9af196766b69221871be78e3\": container with ID starting with 968fe0f23546141ede6d094fb4c361640a2d335b9af196766b69221871be78e3 not found: ID does not exist"
Sep 18 13:52:26.009337 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:26.009306    2586 scope.go:117] "RemoveContainer" containerID="0f023bb75641e724c5216e597428665c64c6cf4313c023cb8ab820f64fbde425"
Sep 18 13:52:26.009629 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:52:26.009608    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f023bb75641e724c5216e597428665c64c6cf4313c023cb8ab820f64fbde425\": container with ID starting with 0f023bb75641e724c5216e597428665c64c6cf4313c023cb8ab820f64fbde425 not found: ID does not exist" containerID="0f023bb75641e724c5216e597428665c64c6cf4313c023cb8ab820f64fbde425"
Sep 18 13:52:26.009692 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:26.009636    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f023bb75641e724c5216e597428665c64c6cf4313c023cb8ab820f64fbde425"} err="failed to get container status \"0f023bb75641e724c5216e597428665c64c6cf4313c023cb8ab820f64fbde425\": rpc error: code = NotFound desc = could not find container \"0f023bb75641e724c5216e597428665c64c6cf4313c023cb8ab820f64fbde425\": container with ID starting with 0f023bb75641e724c5216e597428665c64c6cf4313c023cb8ab820f64fbde425 not found: ID does not exist"
Sep 18 13:52:26.016927 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:26.016887    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"]
Sep 18 13:52:26.023548 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:26.023523    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c50e10-openshift-default-7c564f6846-hplp7"]
Sep 18 13:52:26.849489 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:52:26.849454    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f712e91-6f18-4fe4-9f0b-5550d6a5e368" path="/var/lib/kubelet/pods/5f712e91-6f18-4fe4-9f0b-5550d6a5e368/volumes"
Sep 18 13:53:48.180675 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:53:48.180582    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:54:28.467197 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:54:28.467157    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:54:36.280893 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:54:36.280854    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:54:44.398601 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:54:44.398565    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:55:02.571830 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:55:02.571793    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:55:05.003887 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:55:05.003852    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lcqcw_1cc32441-1189-4113-bf17-2769fa23a747/ovn-acl-logging/0.log"
Sep 18 13:55:05.008881 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:55:05.008857    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lcqcw_1cc32441-1189-4113-bf17-2769fa23a747/ovn-acl-logging/0.log"
Sep 18 13:55:09.172674 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:55:09.172636    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:55:12.302320 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:55:12.302282    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg"]
Sep 18 13:55:12.302735 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:55:12.302480    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg" podUID="ad7f15d2-7e4f-422c-816a-d35e644d573b" containerName="manager" containerID="cri-o://1b778bdc9b39a0494783175dadcc06943c4af77ad749a0e26cae9d90efbac30d" gracePeriod=10
Sep 18 13:55:12.734529 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:55:12.734501    2586 generic.go:356] "Generic (PLEG): container finished" podID="ad7f15d2-7e4f-422c-816a-d35e644d573b" containerID="1b778bdc9b39a0494783175dadcc06943c4af77ad749a0e26cae9d90efbac30d" exitCode=0
Sep 18 13:55:12.734647 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:55:12.734571    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg" event={"ID":"ad7f15d2-7e4f-422c-816a-d35e644d573b","Type":"ContainerDied","Data":"1b778bdc9b39a0494783175dadcc06943c4af77ad749a0e26cae9d90efbac30d"}
Sep 18 13:55:12.734647 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:55:12.734636    2586 scope.go:117] "RemoveContainer" containerID="c68f2889f1bfbb70d939fc952f178820f9af01e4f41124a80e84ed7f5aee0b07"
Sep 18 13:55:12.842952 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:55:12.842925    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg"
Sep 18 13:55:12.918367 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:55:12.918333    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad7f15d2-7e4f-422c-816a-d35e644d573b-extensions-socket-volume\") pod \"ad7f15d2-7e4f-422c-816a-d35e644d573b\" (UID: \"ad7f15d2-7e4f-422c-816a-d35e644d573b\") "
Sep 18 13:55:12.918367 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:55:12.918370    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2l7p\" (UniqueName: \"kubernetes.io/projected/ad7f15d2-7e4f-422c-816a-d35e644d573b-kube-api-access-v2l7p\") pod \"ad7f15d2-7e4f-422c-816a-d35e644d573b\" (UID: \"ad7f15d2-7e4f-422c-816a-d35e644d573b\") "
Sep 18 13:55:12.918736 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:55:12.918711    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7f15d2-7e4f-422c-816a-d35e644d573b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ad7f15d2-7e4f-422c-816a-d35e644d573b" (UID: "ad7f15d2-7e4f-422c-816a-d35e644d573b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:55:12.920640 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:55:12.920611    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad7f15d2-7e4f-422c-816a-d35e644d573b-kube-api-access-v2l7p" (OuterVolumeSpecName: "kube-api-access-v2l7p") pod "ad7f15d2-7e4f-422c-816a-d35e644d573b" (UID: "ad7f15d2-7e4f-422c-816a-d35e644d573b"). InnerVolumeSpecName "kube-api-access-v2l7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:55:13.020003 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:55:13.019969    2586 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad7f15d2-7e4f-422c-816a-d35e644d573b-extensions-socket-volume\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:55:13.020003 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:55:13.019999    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2l7p\" (UniqueName: \"kubernetes.io/projected/ad7f15d2-7e4f-422c-816a-d35e644d573b-kube-api-access-v2l7p\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:55:13.739842 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:55:13.739808    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg" event={"ID":"ad7f15d2-7e4f-422c-816a-d35e644d573b","Type":"ContainerDied","Data":"4ec28290aec1ab92766fc2040bf585ef3738f4c014a60011d4e84beb4ad0498e"}
Sep 18 13:55:13.740332 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:55:13.739867    2586 scope.go:117] "RemoveContainer" containerID="1b778bdc9b39a0494783175dadcc06943c4af77ad749a0e26cae9d90efbac30d"
Sep 18 13:55:13.740332 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:55:13.739818    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg"
Sep 18 13:55:13.763023 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:55:13.762997    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg"]
Sep 18 13:55:13.768766 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:55:13.768745    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7hnrg"]
Sep 18 13:55:14.849714 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:55:14.849637    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad7f15d2-7e4f-422c-816a-d35e644d573b" path="/var/lib/kubelet/pods/ad7f15d2-7e4f-422c-816a-d35e644d573b/volumes"
Sep 18 13:56:13.012163 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012128    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-q4m2d"]
Sep 18 13:56:13.012571 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012441    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e004867-2880-4eda-8770-f1e6fb00eb8d" containerName="istio-proxy"
Sep 18 13:56:13.012571 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012452    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e004867-2880-4eda-8770-f1e6fb00eb8d" containerName="istio-proxy"
Sep 18 13:56:13.012571 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012463    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e004867-2880-4eda-8770-f1e6fb00eb8d" containerName="istio-proxy"
Sep 18 13:56:13.012571 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012473    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e004867-2880-4eda-8770-f1e6fb00eb8d" containerName="istio-proxy"
Sep 18 13:56:13.012571 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012483    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3cb21a3-3d69-4284-a76d-fa0a5a35d064" containerName="istio-proxy"
Sep 18 13:56:13.012571 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012489    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3cb21a3-3d69-4284-a76d-fa0a5a35d064" containerName="istio-proxy"
Sep 18 13:56:13.012571 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012496    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b" containerName="payload-pre-processing"
Sep 18 13:56:13.012571 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012501    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b" containerName="payload-pre-processing"
Sep 18 13:56:13.012571 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012511    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3cb21a3-3d69-4284-a76d-fa0a5a35d064" containerName="istio-proxy"
Sep 18 13:56:13.012571 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012517    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3cb21a3-3d69-4284-a76d-fa0a5a35d064" containerName="istio-proxy"
Sep 18 13:56:13.012571 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012526    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d3fea84-4c32-4c9f-a3aa-f8ab4374076a" containerName="revoke-keys"
Sep 18 13:56:13.012571 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012531    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d3fea84-4c32-4c9f-a3aa-f8ab4374076a" containerName="revoke-keys"
Sep 18 13:56:13.012571 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012540    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f712e91-6f18-4fe4-9f0b-5550d6a5e368" containerName="istio-proxy"
Sep 18 13:56:13.012571 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012549    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f712e91-6f18-4fe4-9f0b-5550d6a5e368" containerName="istio-proxy"
Sep 18 13:56:13.012571 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012559    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad7f15d2-7e4f-422c-816a-d35e644d573b" containerName="manager"
Sep 18 13:56:13.012571 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012564    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad7f15d2-7e4f-422c-816a-d35e644d573b" containerName="manager"
Sep 18 13:56:13.012571 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012575    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65" containerName="istio-proxy"
Sep 18 13:56:13.012571 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012581    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65" containerName="istio-proxy"
Sep 18 13:56:13.013145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012587    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65" containerName="istio-proxy"
Sep 18 13:56:13.013145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012592    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65" containerName="istio-proxy"
Sep 18 13:56:13.013145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012646    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e004867-2880-4eda-8770-f1e6fb00eb8d" containerName="istio-proxy"
Sep 18 13:56:13.013145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012653    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3cb21a3-3d69-4284-a76d-fa0a5a35d064" containerName="istio-proxy"
Sep 18 13:56:13.013145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012659    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ad1326c-f8c5-4d3b-bdd1-0956529c4a7b" containerName="payload-pre-processing"
Sep 18 13:56:13.013145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012665    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65" containerName="istio-proxy"
Sep 18 13:56:13.013145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012673    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e004867-2880-4eda-8770-f1e6fb00eb8d" containerName="istio-proxy"
Sep 18 13:56:13.013145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012678    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d3fea84-4c32-4c9f-a3aa-f8ab4374076a" containerName="revoke-keys"
Sep 18 13:56:13.013145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012683    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3cb21a3-3d69-4284-a76d-fa0a5a35d064" containerName="istio-proxy"
Sep 18 13:56:13.013145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012690    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad7f15d2-7e4f-422c-816a-d35e644d573b" containerName="manager"
Sep 18 13:56:13.013145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012694    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f712e91-6f18-4fe4-9f0b-5550d6a5e368" containerName="istio-proxy"
Sep 18 13:56:13.013145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012702    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f712e91-6f18-4fe4-9f0b-5550d6a5e368" containerName="istio-proxy"
Sep 18 13:56:13.013145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.012711    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad7f15d2-7e4f-422c-816a-d35e644d573b" containerName="manager"
Sep 18 13:56:13.015588 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.015572    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-q4m2d"
Sep 18 13:56:13.018605 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.018583    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sbkdk\""
Sep 18 13:56:13.032392 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.032369    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-q4m2d"]
Sep 18 13:56:13.131503 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.131475    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqzk5\" (UniqueName: \"kubernetes.io/projected/7f0e9d3d-14b8-4a2d-a634-a2ee26719f68-kube-api-access-rqzk5\") pod \"kuadrant-operator-controller-manager-5769f99d9f-q4m2d\" (UID: \"7f0e9d3d-14b8-4a2d-a634-a2ee26719f68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-q4m2d"
Sep 18 13:56:13.131710 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.131607    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f0e9d3d-14b8-4a2d-a634-a2ee26719f68-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-q4m2d\" (UID: \"7f0e9d3d-14b8-4a2d-a634-a2ee26719f68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-q4m2d"
Sep 18 13:56:13.232848 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.232823    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f0e9d3d-14b8-4a2d-a634-a2ee26719f68-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-q4m2d\" (UID: \"7f0e9d3d-14b8-4a2d-a634-a2ee26719f68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-q4m2d"
Sep 18 13:56:13.233059 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.232869    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqzk5\" (UniqueName: \"kubernetes.io/projected/7f0e9d3d-14b8-4a2d-a634-a2ee26719f68-kube-api-access-rqzk5\") pod \"kuadrant-operator-controller-manager-5769f99d9f-q4m2d\" (UID: \"7f0e9d3d-14b8-4a2d-a634-a2ee26719f68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-q4m2d"
Sep 18 13:56:13.233208 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.233188    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7f0e9d3d-14b8-4a2d-a634-a2ee26719f68-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-q4m2d\" (UID: \"7f0e9d3d-14b8-4a2d-a634-a2ee26719f68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-q4m2d"
Sep 18 13:56:13.241603 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.241579    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqzk5\" (UniqueName: \"kubernetes.io/projected/7f0e9d3d-14b8-4a2d-a634-a2ee26719f68-kube-api-access-rqzk5\") pod \"kuadrant-operator-controller-manager-5769f99d9f-q4m2d\" (UID: \"7f0e9d3d-14b8-4a2d-a634-a2ee26719f68\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-q4m2d"
Sep 18 13:56:13.325456 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.325392    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-q4m2d"
Sep 18 13:56:13.612934 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:13.612892    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-q4m2d"]
Sep 18 13:56:13.617593 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:56:13.617560    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0e9d3d_14b8_4a2d_a634_a2ee26719f68.slice/crio-16a31bfa8acc610b210214b7f641ff729c3523e2ac687bd2618ba71b387912d8 WatchSource:0}: Error finding container 16a31bfa8acc610b210214b7f641ff729c3523e2ac687bd2618ba71b387912d8: Status 404 returned error can't find the container with id 16a31bfa8acc610b210214b7f641ff729c3523e2ac687bd2618ba71b387912d8
Sep 18 13:56:14.001767 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:14.001728    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-q4m2d" event={"ID":"7f0e9d3d-14b8-4a2d-a634-a2ee26719f68","Type":"ContainerStarted","Data":"f275d56654e79357d61f280397e41b498e38f4cdb6e19bdfd8f60421e0afdfbc"}
Sep 18 13:56:14.001968 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:14.001782    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-q4m2d" event={"ID":"7f0e9d3d-14b8-4a2d-a634-a2ee26719f68","Type":"ContainerStarted","Data":"16a31bfa8acc610b210214b7f641ff729c3523e2ac687bd2618ba71b387912d8"}
Sep 18 13:56:14.001968 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:14.001871    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-q4m2d"
Sep 18 13:56:14.025082 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:14.025012    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-q4m2d" podStartSLOduration=2.024994202 podStartE2EDuration="2.024994202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:56:14.024127517 +0000 UTC m=+2471.379587342" watchObservedRunningTime="2026-09-18 13:56:14.024994202 +0000 UTC m=+2471.380454028"
Sep 18 13:56:25.008442 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:25.008411    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-q4m2d"
Sep 18 13:56:40.784730 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:40.784691    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:56:46.472403 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:46.472317    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:56:56.562594 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.562561    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"]
Sep 18 13:56:56.562989 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.562931    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f712e91-6f18-4fe4-9f0b-5550d6a5e368" containerName="istio-proxy"
Sep 18 13:56:56.562989 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.562943    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f712e91-6f18-4fe4-9f0b-5550d6a5e368" containerName="istio-proxy"
Sep 18 13:56:56.562989 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.562953    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad7f15d2-7e4f-422c-816a-d35e644d573b" containerName="manager"
Sep 18 13:56:56.562989 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.562959    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad7f15d2-7e4f-422c-816a-d35e644d573b" containerName="manager"
Sep 18 13:56:56.563125 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.563008    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e8cae6a-e1e8-4e4d-88fd-faa5eba60a65" containerName="istio-proxy"
Sep 18 13:56:56.566070 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.566055    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:56:56.568323 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.568297    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-f39fb1e7-kserve-self-signed-certs\""
Sep 18 13:56:56.581200 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.581169    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"]
Sep 18 13:56:56.692633 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.692598    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f00dc942-d058-4990-be07-69771855bfad-home\") pod \"e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv\" (UID: \"f00dc942-d058-4990-be07-69771855bfad\") " pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:56:56.692827 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.692683    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5rl2\" (UniqueName: \"kubernetes.io/projected/f00dc942-d058-4990-be07-69771855bfad-kube-api-access-t5rl2\") pod \"e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv\" (UID: \"f00dc942-d058-4990-be07-69771855bfad\") " pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:56:56.692827 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.692741    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f00dc942-d058-4990-be07-69771855bfad-model-cache\") pod \"e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv\" (UID: \"f00dc942-d058-4990-be07-69771855bfad\") " pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:56:56.692827 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.692797    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00dc942-d058-4990-be07-69771855bfad-tls-certs\") pod \"e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv\" (UID: \"f00dc942-d058-4990-be07-69771855bfad\") " pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:56:56.692996 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.692833    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00dc942-d058-4990-be07-69771855bfad-tmp-dir\") pod \"e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv\" (UID: \"f00dc942-d058-4990-be07-69771855bfad\") " pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:56:56.692996 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.692876    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f00dc942-d058-4990-be07-69771855bfad-dshm\") pod \"e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv\" (UID: \"f00dc942-d058-4990-be07-69771855bfad\") " pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:56:56.793784 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.793747    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00dc942-d058-4990-be07-69771855bfad-tmp-dir\") pod \"e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv\" (UID: \"f00dc942-d058-4990-be07-69771855bfad\") " pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:56:56.793952 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.793788    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f00dc942-d058-4990-be07-69771855bfad-dshm\") pod \"e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv\" (UID: \"f00dc942-d058-4990-be07-69771855bfad\") " pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:56:56.793952 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.793827    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f00dc942-d058-4990-be07-69771855bfad-home\") pod \"e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv\" (UID: \"f00dc942-d058-4990-be07-69771855bfad\") " pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:56:56.793952 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.793868    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5rl2\" (UniqueName: \"kubernetes.io/projected/f00dc942-d058-4990-be07-69771855bfad-kube-api-access-t5rl2\") pod \"e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv\" (UID: \"f00dc942-d058-4990-be07-69771855bfad\") " pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:56:56.793952 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.793886    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f00dc942-d058-4990-be07-69771855bfad-model-cache\") pod \"e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv\" (UID: \"f00dc942-d058-4990-be07-69771855bfad\") " pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:56:56.793952 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.793942    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00dc942-d058-4990-be07-69771855bfad-tls-certs\") pod \"e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv\" (UID: \"f00dc942-d058-4990-be07-69771855bfad\") " pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:56:56.794248 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.794222    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00dc942-d058-4990-be07-69771855bfad-tmp-dir\") pod \"e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv\" (UID: \"f00dc942-d058-4990-be07-69771855bfad\") " pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:56:56.794314 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.794267    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f00dc942-d058-4990-be07-69771855bfad-home\") pod \"e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv\" (UID: \"f00dc942-d058-4990-be07-69771855bfad\") " pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:56:56.794372 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.794334    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f00dc942-d058-4990-be07-69771855bfad-model-cache\") pod \"e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv\" (UID: \"f00dc942-d058-4990-be07-69771855bfad\") " pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:56:56.796289 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.796262    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f00dc942-d058-4990-be07-69771855bfad-dshm\") pod \"e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv\" (UID: \"f00dc942-d058-4990-be07-69771855bfad\") " pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:56:56.796443 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.796427    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00dc942-d058-4990-be07-69771855bfad-tls-certs\") pod \"e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv\" (UID: \"f00dc942-d058-4990-be07-69771855bfad\") " pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:56:56.801862 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.801845    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5rl2\" (UniqueName: \"kubernetes.io/projected/f00dc942-d058-4990-be07-69771855bfad-kube-api-access-t5rl2\") pod \"e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv\" (UID: \"f00dc942-d058-4990-be07-69771855bfad\") " pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:56:56.875015 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:56.874951    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:56:57.154803 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:57.154769    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"]
Sep 18 13:56:57.157871 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:56:57.157841    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf00dc942_d058_4990_be07_69771855bfad.slice/crio-b0e9875b3e34459d91878b0ecef8fb559029aad6df3cdce69299d4e2b484cc7a WatchSource:0}: Error finding container b0e9875b3e34459d91878b0ecef8fb559029aad6df3cdce69299d4e2b484cc7a: Status 404 returned error can't find the container with id b0e9875b3e34459d91878b0ecef8fb559029aad6df3cdce69299d4e2b484cc7a
Sep 18 13:56:57.193044 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:57.193010    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv" event={"ID":"f00dc942-d058-4990-be07-69771855bfad","Type":"ContainerStarted","Data":"b0e9875b3e34459d91878b0ecef8fb559029aad6df3cdce69299d4e2b484cc7a"}
Sep 18 13:56:58.198664 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:58.198626    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv" event={"ID":"f00dc942-d058-4990-be07-69771855bfad","Type":"ContainerStarted","Data":"58c12974644988b90a376fcc3b3817b58961cdcdd08a2d3c073d08b5f0ac614f"}
Sep 18 13:56:58.224258 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:58.224207    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv" podStartSLOduration=2.224192612 podStartE2EDuration="2.224192612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:56:58.220551141 +0000 UTC m=+2515.576010966" watchObservedRunningTime="2026-09-18 13:56:58.224192612 +0000 UTC m=+2515.579652437"
Sep 18 13:56:59.080142 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:56:59.080108    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:57:05.330305 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:05.330269    2586 scope.go:117] "RemoveContainer" containerID="32785454ee30053b82575855381a1978306b8387bb17ead253bf0a6bc03cb702"
Sep 18 13:57:06.876014 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:06.875963    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:57:06.876014 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:06.876023    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:57:06.888874 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:06.888844    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:57:06.990443 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:06.990413    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"]
Sep 18 13:57:06.994863 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:06.994846    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:57:06.996861 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:06.996843    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-f39fb1e7-kserve-self-signed-certs\""
Sep 18 13:57:07.007219 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:07.007197    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"]
Sep 18 13:57:07.185477 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:07.185427    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa05996b-503e-4d86-a55d-b4d698cd96ca-tls-certs\") pod \"e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp\" (UID: \"aa05996b-503e-4d86-a55d-b4d698cd96ca\") " pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:57:07.185477 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:07.185479    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa05996b-503e-4d86-a55d-b4d698cd96ca-dshm\") pod \"e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp\" (UID: \"aa05996b-503e-4d86-a55d-b4d698cd96ca\") " pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:57:07.185815 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:07.185593    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa05996b-503e-4d86-a55d-b4d698cd96ca-home\") pod \"e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp\" (UID: \"aa05996b-503e-4d86-a55d-b4d698cd96ca\") " pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:57:07.185815 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:07.185663    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fckdw\" (UniqueName: \"kubernetes.io/projected/aa05996b-503e-4d86-a55d-b4d698cd96ca-kube-api-access-fckdw\") pod \"e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp\" (UID: \"aa05996b-503e-4d86-a55d-b4d698cd96ca\") " pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:57:07.185815 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:07.185702    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa05996b-503e-4d86-a55d-b4d698cd96ca-model-cache\") pod \"e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp\" (UID: \"aa05996b-503e-4d86-a55d-b4d698cd96ca\") " pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:57:07.185815 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:07.185723    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa05996b-503e-4d86-a55d-b4d698cd96ca-tmp-dir\") pod \"e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp\" (UID: \"aa05996b-503e-4d86-a55d-b4d698cd96ca\") " pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:57:07.253177 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:07.253149    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:57:07.286707 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:07.286676    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa05996b-503e-4d86-a55d-b4d698cd96ca-home\") pod \"e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp\" (UID: \"aa05996b-503e-4d86-a55d-b4d698cd96ca\") " pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:57:07.286887 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:07.286724    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fckdw\" (UniqueName: \"kubernetes.io/projected/aa05996b-503e-4d86-a55d-b4d698cd96ca-kube-api-access-fckdw\") pod \"e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp\" (UID: \"aa05996b-503e-4d86-a55d-b4d698cd96ca\") " pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:57:07.286887 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:07.286752    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa05996b-503e-4d86-a55d-b4d698cd96ca-model-cache\") pod \"e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp\" (UID: \"aa05996b-503e-4d86-a55d-b4d698cd96ca\") " pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:57:07.286887 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:07.286768    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa05996b-503e-4d86-a55d-b4d698cd96ca-tmp-dir\") pod \"e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp\" (UID: \"aa05996b-503e-4d86-a55d-b4d698cd96ca\") " pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:57:07.287506 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:07.286997    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa05996b-503e-4d86-a55d-b4d698cd96ca-tls-certs\") pod \"e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp\" (UID: \"aa05996b-503e-4d86-a55d-b4d698cd96ca\") " pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:57:07.287613 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:07.287540    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa05996b-503e-4d86-a55d-b4d698cd96ca-dshm\") pod \"e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp\" (UID: \"aa05996b-503e-4d86-a55d-b4d698cd96ca\") " pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:57:07.287613 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:07.287131    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa05996b-503e-4d86-a55d-b4d698cd96ca-home\") pod \"e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp\" (UID: \"aa05996b-503e-4d86-a55d-b4d698cd96ca\") " pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:57:07.287613 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:07.287203    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa05996b-503e-4d86-a55d-b4d698cd96ca-model-cache\") pod \"e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp\" (UID: \"aa05996b-503e-4d86-a55d-b4d698cd96ca\") " pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:57:07.287756 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:07.287157    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa05996b-503e-4d86-a55d-b4d698cd96ca-tmp-dir\") pod \"e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp\" (UID: \"aa05996b-503e-4d86-a55d-b4d698cd96ca\") " pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:57:07.289675 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:07.289638    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa05996b-503e-4d86-a55d-b4d698cd96ca-tls-certs\") pod \"e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp\" (UID: \"aa05996b-503e-4d86-a55d-b4d698cd96ca\") " pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:57:07.289859 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:07.289840    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa05996b-503e-4d86-a55d-b4d698cd96ca-dshm\") pod \"e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp\" (UID: \"aa05996b-503e-4d86-a55d-b4d698cd96ca\") " pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:57:07.297477 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:07.297454    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fckdw\" (UniqueName: \"kubernetes.io/projected/aa05996b-503e-4d86-a55d-b4d698cd96ca-kube-api-access-fckdw\") pod \"e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp\" (UID: \"aa05996b-503e-4d86-a55d-b4d698cd96ca\") " pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:57:07.303738 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:07.303485    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:57:07.586371 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:07.586280    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"]
Sep 18 13:57:07.590516 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:57:07.590488    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa05996b_503e_4d86_a55d_b4d698cd96ca.slice/crio-cb604516f972e80859fa8cd8efb61d288f0da343ee570ddd367b9e59ae5cea9f WatchSource:0}: Error finding container cb604516f972e80859fa8cd8efb61d288f0da343ee570ddd367b9e59ae5cea9f: Status 404 returned error can't find the container with id cb604516f972e80859fa8cd8efb61d288f0da343ee570ddd367b9e59ae5cea9f
Sep 18 13:57:08.248267 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:08.248229    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp" event={"ID":"aa05996b-503e-4d86-a55d-b4d698cd96ca","Type":"ContainerStarted","Data":"15ed74202037de97110ce9e50901286813edba782cd753e676f4d8c221c029d3"}
Sep 18 13:57:08.248267 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:08.248272    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp" event={"ID":"aa05996b-503e-4d86-a55d-b4d698cd96ca","Type":"ContainerStarted","Data":"cb604516f972e80859fa8cd8efb61d288f0da343ee570ddd367b9e59ae5cea9f"}
Sep 18 13:57:08.272284 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:08.272227    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp" podStartSLOduration=2.272208997 podStartE2EDuration="2.272208997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:57:08.271028263 +0000 UTC m=+2525.626488088" watchObservedRunningTime="2026-09-18 13:57:08.272208997 +0000 UTC m=+2525.627668824"
Sep 18 13:57:10.074513 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:10.074478    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:57:17.305032 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:17.304980    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:57:17.305032 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:17.305039    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:57:17.319170 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:17.319137    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:57:18.308323 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:18.308291    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:57:48.743033 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:48.742995    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"]
Sep 18 13:57:49.561226 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:49.561192    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"]
Sep 18 13:57:52.073216 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:57:52.073175    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:58:03.763155 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:03.763103    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv" podUID="f00dc942-d058-4990-be07-69771855bfad" containerName="main" containerID="cri-o://58c12974644988b90a376fcc3b3817b58961cdcdd08a2d3c073d08b5f0ac614f" gracePeriod=45
Sep 18 13:58:04.014736 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.014678    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:58:04.096812 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.096786    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f00dc942-d058-4990-be07-69771855bfad-home\") pod \"f00dc942-d058-4990-be07-69771855bfad\" (UID: \"f00dc942-d058-4990-be07-69771855bfad\") "
Sep 18 13:58:04.096812 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.096814    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00dc942-d058-4990-be07-69771855bfad-tls-certs\") pod \"f00dc942-d058-4990-be07-69771855bfad\" (UID: \"f00dc942-d058-4990-be07-69771855bfad\") "
Sep 18 13:58:04.097054 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.096867    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5rl2\" (UniqueName: \"kubernetes.io/projected/f00dc942-d058-4990-be07-69771855bfad-kube-api-access-t5rl2\") pod \"f00dc942-d058-4990-be07-69771855bfad\" (UID: \"f00dc942-d058-4990-be07-69771855bfad\") "
Sep 18 13:58:04.097054 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.096885    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00dc942-d058-4990-be07-69771855bfad-tmp-dir\") pod \"f00dc942-d058-4990-be07-69771855bfad\" (UID: \"f00dc942-d058-4990-be07-69771855bfad\") "
Sep 18 13:58:04.097054 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.096921    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f00dc942-d058-4990-be07-69771855bfad-model-cache\") pod \"f00dc942-d058-4990-be07-69771855bfad\" (UID: \"f00dc942-d058-4990-be07-69771855bfad\") "
Sep 18 13:58:04.097054 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.096960    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f00dc942-d058-4990-be07-69771855bfad-dshm\") pod \"f00dc942-d058-4990-be07-69771855bfad\" (UID: \"f00dc942-d058-4990-be07-69771855bfad\") "
Sep 18 13:58:04.097317 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.097055    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00dc942-d058-4990-be07-69771855bfad-home" (OuterVolumeSpecName: "home") pod "f00dc942-d058-4990-be07-69771855bfad" (UID: "f00dc942-d058-4990-be07-69771855bfad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:58:04.097317 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.097192    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00dc942-d058-4990-be07-69771855bfad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f00dc942-d058-4990-be07-69771855bfad" (UID: "f00dc942-d058-4990-be07-69771855bfad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:58:04.097317 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.097280    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00dc942-d058-4990-be07-69771855bfad-model-cache" (OuterVolumeSpecName: "model-cache") pod "f00dc942-d058-4990-be07-69771855bfad" (UID: "f00dc942-d058-4990-be07-69771855bfad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:58:04.097438 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.097325    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f00dc942-d058-4990-be07-69771855bfad-tmp-dir\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:58:04.097438 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.097345    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f00dc942-d058-4990-be07-69771855bfad-home\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:58:04.099133 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.099110    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f00dc942-d058-4990-be07-69771855bfad-dshm" (OuterVolumeSpecName: "dshm") pod "f00dc942-d058-4990-be07-69771855bfad" (UID: "f00dc942-d058-4990-be07-69771855bfad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:58:04.099204 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.099190    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00dc942-d058-4990-be07-69771855bfad-kube-api-access-t5rl2" (OuterVolumeSpecName: "kube-api-access-t5rl2") pod "f00dc942-d058-4990-be07-69771855bfad" (UID: "f00dc942-d058-4990-be07-69771855bfad"). InnerVolumeSpecName "kube-api-access-t5rl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:58:04.099245 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.099227    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f00dc942-d058-4990-be07-69771855bfad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f00dc942-d058-4990-be07-69771855bfad" (UID: "f00dc942-d058-4990-be07-69771855bfad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:58:04.198721 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.198693    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5rl2\" (UniqueName: \"kubernetes.io/projected/f00dc942-d058-4990-be07-69771855bfad-kube-api-access-t5rl2\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:58:04.198721 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.198720    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f00dc942-d058-4990-be07-69771855bfad-model-cache\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:58:04.198721 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.198730    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f00dc942-d058-4990-be07-69771855bfad-dshm\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:58:04.198980 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.198738    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f00dc942-d058-4990-be07-69771855bfad-tls-certs\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:58:04.505584 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.505550    2586 generic.go:356] "Generic (PLEG): container finished" podID="f00dc942-d058-4990-be07-69771855bfad" containerID="58c12974644988b90a376fcc3b3817b58961cdcdd08a2d3c073d08b5f0ac614f" exitCode=0
Sep 18 13:58:04.505766 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.505595    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"
Sep 18 13:58:04.505766 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.505635    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv" event={"ID":"f00dc942-d058-4990-be07-69771855bfad","Type":"ContainerDied","Data":"58c12974644988b90a376fcc3b3817b58961cdcdd08a2d3c073d08b5f0ac614f"}
Sep 18 13:58:04.505766 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.505678    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv" event={"ID":"f00dc942-d058-4990-be07-69771855bfad","Type":"ContainerDied","Data":"b0e9875b3e34459d91878b0ecef8fb559029aad6df3cdce69299d4e2b484cc7a"}
Sep 18 13:58:04.505766 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.505695    2586 scope.go:117] "RemoveContainer" containerID="58c12974644988b90a376fcc3b3817b58961cdcdd08a2d3c073d08b5f0ac614f"
Sep 18 13:58:04.514952 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.514929    2586 scope.go:117] "RemoveContainer" containerID="58c12974644988b90a376fcc3b3817b58961cdcdd08a2d3c073d08b5f0ac614f"
Sep 18 13:58:04.515219 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:58:04.515200    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58c12974644988b90a376fcc3b3817b58961cdcdd08a2d3c073d08b5f0ac614f\": container with ID starting with 58c12974644988b90a376fcc3b3817b58961cdcdd08a2d3c073d08b5f0ac614f not found: ID does not exist" containerID="58c12974644988b90a376fcc3b3817b58961cdcdd08a2d3c073d08b5f0ac614f"
Sep 18 13:58:04.515286 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.515227    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58c12974644988b90a376fcc3b3817b58961cdcdd08a2d3c073d08b5f0ac614f"} err="failed to get container status \"58c12974644988b90a376fcc3b3817b58961cdcdd08a2d3c073d08b5f0ac614f\": rpc error: code = NotFound desc = could not find container \"58c12974644988b90a376fcc3b3817b58961cdcdd08a2d3c073d08b5f0ac614f\": container with ID starting with 58c12974644988b90a376fcc3b3817b58961cdcdd08a2d3c073d08b5f0ac614f not found: ID does not exist"
Sep 18 13:58:04.565278 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.565253    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"]
Sep 18 13:58:04.570359 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.570336    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-f39fb1e7-kserve-75fb9787-x5qrv"]
Sep 18 13:58:04.580671 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.580638    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp" podUID="aa05996b-503e-4d86-a55d-b4d698cd96ca" containerName="main" containerID="cri-o://15ed74202037de97110ce9e50901286813edba782cd753e676f4d8c221c029d3" gracePeriod=45
Sep 18 13:58:04.822023 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.822000    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:58:04.849550 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.849522    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f00dc942-d058-4990-be07-69771855bfad" path="/var/lib/kubelet/pods/f00dc942-d058-4990-be07-69771855bfad/volumes"
Sep 18 13:58:04.904174 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.904142    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa05996b-503e-4d86-a55d-b4d698cd96ca-home\") pod \"aa05996b-503e-4d86-a55d-b4d698cd96ca\" (UID: \"aa05996b-503e-4d86-a55d-b4d698cd96ca\") "
Sep 18 13:58:04.904341 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.904188    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa05996b-503e-4d86-a55d-b4d698cd96ca-dshm\") pod \"aa05996b-503e-4d86-a55d-b4d698cd96ca\" (UID: \"aa05996b-503e-4d86-a55d-b4d698cd96ca\") "
Sep 18 13:58:04.904341 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.904218    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa05996b-503e-4d86-a55d-b4d698cd96ca-tls-certs\") pod \"aa05996b-503e-4d86-a55d-b4d698cd96ca\" (UID: \"aa05996b-503e-4d86-a55d-b4d698cd96ca\") "
Sep 18 13:58:04.904341 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.904260    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fckdw\" (UniqueName: \"kubernetes.io/projected/aa05996b-503e-4d86-a55d-b4d698cd96ca-kube-api-access-fckdw\") pod \"aa05996b-503e-4d86-a55d-b4d698cd96ca\" (UID: \"aa05996b-503e-4d86-a55d-b4d698cd96ca\") "
Sep 18 13:58:04.904341 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.904323    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa05996b-503e-4d86-a55d-b4d698cd96ca-tmp-dir\") pod \"aa05996b-503e-4d86-a55d-b4d698cd96ca\" (UID: \"aa05996b-503e-4d86-a55d-b4d698cd96ca\") "
Sep 18 13:58:04.904571 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.904411    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa05996b-503e-4d86-a55d-b4d698cd96ca-model-cache\") pod \"aa05996b-503e-4d86-a55d-b4d698cd96ca\" (UID: \"aa05996b-503e-4d86-a55d-b4d698cd96ca\") "
Sep 18 13:58:04.904571 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.904462    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa05996b-503e-4d86-a55d-b4d698cd96ca-home" (OuterVolumeSpecName: "home") pod "aa05996b-503e-4d86-a55d-b4d698cd96ca" (UID: "aa05996b-503e-4d86-a55d-b4d698cd96ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:58:04.904708 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.904684    2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa05996b-503e-4d86-a55d-b4d698cd96ca-home\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:58:04.904789 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.904769    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa05996b-503e-4d86-a55d-b4d698cd96ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa05996b-503e-4d86-a55d-b4d698cd96ca" (UID: "aa05996b-503e-4d86-a55d-b4d698cd96ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:58:04.904899 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.904855    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa05996b-503e-4d86-a55d-b4d698cd96ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa05996b-503e-4d86-a55d-b4d698cd96ca" (UID: "aa05996b-503e-4d86-a55d-b4d698cd96ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:58:04.906500 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.906467    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa05996b-503e-4d86-a55d-b4d698cd96ca-dshm" (OuterVolumeSpecName: "dshm") pod "aa05996b-503e-4d86-a55d-b4d698cd96ca" (UID: "aa05996b-503e-4d86-a55d-b4d698cd96ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 13:58:04.907131 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.907105    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa05996b-503e-4d86-a55d-b4d698cd96ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa05996b-503e-4d86-a55d-b4d698cd96ca" (UID: "aa05996b-503e-4d86-a55d-b4d698cd96ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:58:04.907237 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:04.907151    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa05996b-503e-4d86-a55d-b4d698cd96ca-kube-api-access-fckdw" (OuterVolumeSpecName: "kube-api-access-fckdw") pod "aa05996b-503e-4d86-a55d-b4d698cd96ca" (UID: "aa05996b-503e-4d86-a55d-b4d698cd96ca"). InnerVolumeSpecName "kube-api-access-fckdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:58:05.005484 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:05.005454    2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa05996b-503e-4d86-a55d-b4d698cd96ca-dshm\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:58:05.005484 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:05.005485    2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa05996b-503e-4d86-a55d-b4d698cd96ca-tls-certs\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:58:05.005659 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:05.005501    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fckdw\" (UniqueName: \"kubernetes.io/projected/aa05996b-503e-4d86-a55d-b4d698cd96ca-kube-api-access-fckdw\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:58:05.005659 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:05.005509    2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa05996b-503e-4d86-a55d-b4d698cd96ca-tmp-dir\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:58:05.005659 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:05.005518    2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa05996b-503e-4d86-a55d-b4d698cd96ca-model-cache\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:58:05.290752 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:05.290716    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:58:05.349086 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:05.349052    2586 scope.go:117] "RemoveContainer" containerID="301a95a191478b7c6e4ca4bf6e2a8874e25366a6b38cf661fcccc1f092cdfbb0"
Sep 18 13:58:05.511630 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:05.511592    2586 generic.go:356] "Generic (PLEG): container finished" podID="aa05996b-503e-4d86-a55d-b4d698cd96ca" containerID="15ed74202037de97110ce9e50901286813edba782cd753e676f4d8c221c029d3" exitCode=0
Sep 18 13:58:05.511829 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:05.511641    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp" event={"ID":"aa05996b-503e-4d86-a55d-b4d698cd96ca","Type":"ContainerDied","Data":"15ed74202037de97110ce9e50901286813edba782cd753e676f4d8c221c029d3"}
Sep 18 13:58:05.511829 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:05.511654    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"
Sep 18 13:58:05.511829 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:05.511671    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp" event={"ID":"aa05996b-503e-4d86-a55d-b4d698cd96ca","Type":"ContainerDied","Data":"cb604516f972e80859fa8cd8efb61d288f0da343ee570ddd367b9e59ae5cea9f"}
Sep 18 13:58:05.511829 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:05.511693    2586 scope.go:117] "RemoveContainer" containerID="15ed74202037de97110ce9e50901286813edba782cd753e676f4d8c221c029d3"
Sep 18 13:58:05.520488 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:05.520466    2586 scope.go:117] "RemoveContainer" containerID="15ed74202037de97110ce9e50901286813edba782cd753e676f4d8c221c029d3"
Sep 18 13:58:05.520758 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:58:05.520739    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15ed74202037de97110ce9e50901286813edba782cd753e676f4d8c221c029d3\": container with ID starting with 15ed74202037de97110ce9e50901286813edba782cd753e676f4d8c221c029d3 not found: ID does not exist" containerID="15ed74202037de97110ce9e50901286813edba782cd753e676f4d8c221c029d3"
Sep 18 13:58:05.520818 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:05.520768    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15ed74202037de97110ce9e50901286813edba782cd753e676f4d8c221c029d3"} err="failed to get container status \"15ed74202037de97110ce9e50901286813edba782cd753e676f4d8c221c029d3\": rpc error: code = NotFound desc = could not find container \"15ed74202037de97110ce9e50901286813edba782cd753e676f4d8c221c029d3\": container with ID starting with 15ed74202037de97110ce9e50901286813edba782cd753e676f4d8c221c029d3 not found: ID does not exist"
Sep 18 13:58:05.534576 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:05.534549    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"]
Sep 18 13:58:05.539163 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:05.539142    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-f39fb1e7-kserve-b445b586c-5cxpp"]
Sep 18 13:58:06.850342 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:06.850297    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa05996b-503e-4d86-a55d-b4d698cd96ca" path="/var/lib/kubelet/pods/aa05996b-503e-4d86-a55d-b4d698cd96ca/volumes"
Sep 18 13:58:12.057401 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.057365    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5b89985555-7rwhr"]
Sep 18 13:58:12.057891 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.057831    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa05996b-503e-4d86-a55d-b4d698cd96ca" containerName="main"
Sep 18 13:58:12.057891 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.057849    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa05996b-503e-4d86-a55d-b4d698cd96ca" containerName="main"
Sep 18 13:58:12.057891 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.057877    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f00dc942-d058-4990-be07-69771855bfad" containerName="main"
Sep 18 13:58:12.057891 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.057887    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f00dc942-d058-4990-be07-69771855bfad" containerName="main"
Sep 18 13:58:12.058167 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.057991    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f00dc942-d058-4990-be07-69771855bfad" containerName="main"
Sep 18 13:58:12.058167 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.058006    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa05996b-503e-4d86-a55d-b4d698cd96ca" containerName="main"
Sep 18 13:58:12.062635 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.062609    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b89985555-7rwhr"
Sep 18 13:58:12.068646 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.068610    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 13:58:12.068792 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.068677    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 18 13:58:12.068792 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.068671    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 13:58:12.068792 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.068677    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ffxkp\""
Sep 18 13:58:12.068977 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.068964    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 13:58:12.086206 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.086175    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b89985555-7rwhr"]
Sep 18 13:58:12.164006 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.163968    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w4wz\" (UniqueName: \"kubernetes.io/projected/f97f4c0c-7cdd-43e1-9012-5cf42b610b7e-kube-api-access-7w4wz\") pod \"maas-api-5b89985555-7rwhr\" (UID: \"f97f4c0c-7cdd-43e1-9012-5cf42b610b7e\") " pod="odh-ai-gateway-infra/maas-api-5b89985555-7rwhr"
Sep 18 13:58:12.164169 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.164073    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f97f4c0c-7cdd-43e1-9012-5cf42b610b7e-metrics-certs\") pod \"maas-api-5b89985555-7rwhr\" (UID: \"f97f4c0c-7cdd-43e1-9012-5cf42b610b7e\") " pod="odh-ai-gateway-infra/maas-api-5b89985555-7rwhr"
Sep 18 13:58:12.164169 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.164104    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f97f4c0c-7cdd-43e1-9012-5cf42b610b7e-maas-api-tls\") pod \"maas-api-5b89985555-7rwhr\" (UID: \"f97f4c0c-7cdd-43e1-9012-5cf42b610b7e\") " pod="odh-ai-gateway-infra/maas-api-5b89985555-7rwhr"
Sep 18 13:58:12.164169 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.164163    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f97f4c0c-7cdd-43e1-9012-5cf42b610b7e-openshift-service-ca\") pod \"maas-api-5b89985555-7rwhr\" (UID: \"f97f4c0c-7cdd-43e1-9012-5cf42b610b7e\") " pod="odh-ai-gateway-infra/maas-api-5b89985555-7rwhr"
Sep 18 13:58:12.191649 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.191618    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:58:12.265248 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.265214    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w4wz\" (UniqueName: \"kubernetes.io/projected/f97f4c0c-7cdd-43e1-9012-5cf42b610b7e-kube-api-access-7w4wz\") pod \"maas-api-5b89985555-7rwhr\" (UID: \"f97f4c0c-7cdd-43e1-9012-5cf42b610b7e\") " pod="odh-ai-gateway-infra/maas-api-5b89985555-7rwhr"
Sep 18 13:58:12.265421 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.265298    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f97f4c0c-7cdd-43e1-9012-5cf42b610b7e-metrics-certs\") pod \"maas-api-5b89985555-7rwhr\" (UID: \"f97f4c0c-7cdd-43e1-9012-5cf42b610b7e\") " pod="odh-ai-gateway-infra/maas-api-5b89985555-7rwhr"
Sep 18 13:58:12.265421 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.265326    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f97f4c0c-7cdd-43e1-9012-5cf42b610b7e-maas-api-tls\") pod \"maas-api-5b89985555-7rwhr\" (UID: \"f97f4c0c-7cdd-43e1-9012-5cf42b610b7e\") " pod="odh-ai-gateway-infra/maas-api-5b89985555-7rwhr"
Sep 18 13:58:12.265421 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.265375    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f97f4c0c-7cdd-43e1-9012-5cf42b610b7e-openshift-service-ca\") pod \"maas-api-5b89985555-7rwhr\" (UID: \"f97f4c0c-7cdd-43e1-9012-5cf42b610b7e\") " pod="odh-ai-gateway-infra/maas-api-5b89985555-7rwhr"
Sep 18 13:58:12.266031 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.266007    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f97f4c0c-7cdd-43e1-9012-5cf42b610b7e-openshift-service-ca\") pod \"maas-api-5b89985555-7rwhr\" (UID: \"f97f4c0c-7cdd-43e1-9012-5cf42b610b7e\") " pod="odh-ai-gateway-infra/maas-api-5b89985555-7rwhr"
Sep 18 13:58:12.267843 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.267821    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f97f4c0c-7cdd-43e1-9012-5cf42b610b7e-metrics-certs\") pod \"maas-api-5b89985555-7rwhr\" (UID: \"f97f4c0c-7cdd-43e1-9012-5cf42b610b7e\") " pod="odh-ai-gateway-infra/maas-api-5b89985555-7rwhr"
Sep 18 13:58:12.267975 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.267861    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f97f4c0c-7cdd-43e1-9012-5cf42b610b7e-maas-api-tls\") pod \"maas-api-5b89985555-7rwhr\" (UID: \"f97f4c0c-7cdd-43e1-9012-5cf42b610b7e\") " pod="odh-ai-gateway-infra/maas-api-5b89985555-7rwhr"
Sep 18 13:58:12.273491 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.273461    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w4wz\" (UniqueName: \"kubernetes.io/projected/f97f4c0c-7cdd-43e1-9012-5cf42b610b7e-kube-api-access-7w4wz\") pod \"maas-api-5b89985555-7rwhr\" (UID: \"f97f4c0c-7cdd-43e1-9012-5cf42b610b7e\") " pod="odh-ai-gateway-infra/maas-api-5b89985555-7rwhr"
Sep 18 13:58:12.372723 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.372632    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b89985555-7rwhr"
Sep 18 13:58:12.646770 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.646735    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b89985555-7rwhr"]
Sep 18 13:58:12.649344 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:58:12.649316    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97f4c0c_7cdd_43e1_9012_5cf42b610b7e.slice/crio-0b65f4921186a24d3bd3c7d633c4c428bbcbdfdab7b0837bd96180922da44a65 WatchSource:0}: Error finding container 0b65f4921186a24d3bd3c7d633c4c428bbcbdfdab7b0837bd96180922da44a65: Status 404 returned error can't find the container with id 0b65f4921186a24d3bd3c7d633c4c428bbcbdfdab7b0837bd96180922da44a65
Sep 18 13:58:12.650655 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:12.650633    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 13:58:13.552532 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:13.552452    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b89985555-7rwhr" event={"ID":"f97f4c0c-7cdd-43e1-9012-5cf42b610b7e","Type":"ContainerStarted","Data":"0b65f4921186a24d3bd3c7d633c4c428bbcbdfdab7b0837bd96180922da44a65"}
Sep 18 13:58:15.562263 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:15.562227    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b89985555-7rwhr" event={"ID":"f97f4c0c-7cdd-43e1-9012-5cf42b610b7e","Type":"ContainerStarted","Data":"e08cac5ef8d346cb99cdbda7b707c6f813397936ceaa5a0ab932e8303de8e40a"}
Sep 18 13:58:15.562698 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:15.562435    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5b89985555-7rwhr"
Sep 18 13:58:15.579492 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:15.579450    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5b89985555-7rwhr" podStartSLOduration=1.26744006 podStartE2EDuration="3.57943559s" totalImagesPullingTime="2.31199553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:58:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:15.57795555 +0000 UTC m=+2592.933415367" watchObservedRunningTime="2026-09-18 13:58:15.57943559 +0000 UTC m=+2592.934895416"
Sep 18 13:58:19.509121 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.509076    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b89985555-7rwhr"]
Sep 18 13:58:19.509630 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.509299    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5b89985555-7rwhr" podUID="f97f4c0c-7cdd-43e1-9012-5cf42b610b7e" containerName="maas-api" containerID="cri-o://e08cac5ef8d346cb99cdbda7b707c6f813397936ceaa5a0ab932e8303de8e40a" gracePeriod=30
Sep 18 13:58:19.546985 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.546952    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-76cf6447c5-w4rw6"]
Sep 18 13:58:19.550175 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.550159    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76cf6447c5-w4rw6"
Sep 18 13:58:19.562259 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.562231    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76cf6447c5-w4rw6"]
Sep 18 13:58:19.581986 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.581927    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:58:19.635887 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.635856    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae23680-f7b9-40db-9826-78ec43a1b8b9-openshift-service-ca\") pod \"maas-api-76cf6447c5-w4rw6\" (UID: \"3ae23680-f7b9-40db-9826-78ec43a1b8b9\") " pod="odh-ai-gateway-infra/maas-api-76cf6447c5-w4rw6"
Sep 18 13:58:19.636114 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.635932    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ae23680-f7b9-40db-9826-78ec43a1b8b9-maas-api-tls\") pod \"maas-api-76cf6447c5-w4rw6\" (UID: \"3ae23680-f7b9-40db-9826-78ec43a1b8b9\") " pod="odh-ai-gateway-infra/maas-api-76cf6447c5-w4rw6"
Sep 18 13:58:19.636114 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.635988    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxw2s\" (UniqueName: \"kubernetes.io/projected/3ae23680-f7b9-40db-9826-78ec43a1b8b9-kube-api-access-xxw2s\") pod \"maas-api-76cf6447c5-w4rw6\" (UID: \"3ae23680-f7b9-40db-9826-78ec43a1b8b9\") " pod="odh-ai-gateway-infra/maas-api-76cf6447c5-w4rw6"
Sep 18 13:58:19.636114 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.636040    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ae23680-f7b9-40db-9826-78ec43a1b8b9-metrics-certs\") pod \"maas-api-76cf6447c5-w4rw6\" (UID: \"3ae23680-f7b9-40db-9826-78ec43a1b8b9\") " pod="odh-ai-gateway-infra/maas-api-76cf6447c5-w4rw6"
Sep 18 13:58:19.737280 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.737241    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae23680-f7b9-40db-9826-78ec43a1b8b9-openshift-service-ca\") pod \"maas-api-76cf6447c5-w4rw6\" (UID: \"3ae23680-f7b9-40db-9826-78ec43a1b8b9\") " pod="odh-ai-gateway-infra/maas-api-76cf6447c5-w4rw6"
Sep 18 13:58:19.737465 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.737292    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ae23680-f7b9-40db-9826-78ec43a1b8b9-maas-api-tls\") pod \"maas-api-76cf6447c5-w4rw6\" (UID: \"3ae23680-f7b9-40db-9826-78ec43a1b8b9\") " pod="odh-ai-gateway-infra/maas-api-76cf6447c5-w4rw6"
Sep 18 13:58:19.737465 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.737342    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxw2s\" (UniqueName: \"kubernetes.io/projected/3ae23680-f7b9-40db-9826-78ec43a1b8b9-kube-api-access-xxw2s\") pod \"maas-api-76cf6447c5-w4rw6\" (UID: \"3ae23680-f7b9-40db-9826-78ec43a1b8b9\") " pod="odh-ai-gateway-infra/maas-api-76cf6447c5-w4rw6"
Sep 18 13:58:19.737465 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.737373    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ae23680-f7b9-40db-9826-78ec43a1b8b9-metrics-certs\") pod \"maas-api-76cf6447c5-w4rw6\" (UID: \"3ae23680-f7b9-40db-9826-78ec43a1b8b9\") " pod="odh-ai-gateway-infra/maas-api-76cf6447c5-w4rw6"
Sep 18 13:58:19.738147 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.738058    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae23680-f7b9-40db-9826-78ec43a1b8b9-openshift-service-ca\") pod \"maas-api-76cf6447c5-w4rw6\" (UID: \"3ae23680-f7b9-40db-9826-78ec43a1b8b9\") " pod="odh-ai-gateway-infra/maas-api-76cf6447c5-w4rw6"
Sep 18 13:58:19.740050 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.740029    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ae23680-f7b9-40db-9826-78ec43a1b8b9-metrics-certs\") pod \"maas-api-76cf6447c5-w4rw6\" (UID: \"3ae23680-f7b9-40db-9826-78ec43a1b8b9\") " pod="odh-ai-gateway-infra/maas-api-76cf6447c5-w4rw6"
Sep 18 13:58:19.740138 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.740120    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3ae23680-f7b9-40db-9826-78ec43a1b8b9-maas-api-tls\") pod \"maas-api-76cf6447c5-w4rw6\" (UID: \"3ae23680-f7b9-40db-9826-78ec43a1b8b9\") " pod="odh-ai-gateway-infra/maas-api-76cf6447c5-w4rw6"
Sep 18 13:58:19.745471 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.745454    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxw2s\" (UniqueName: \"kubernetes.io/projected/3ae23680-f7b9-40db-9826-78ec43a1b8b9-kube-api-access-xxw2s\") pod \"maas-api-76cf6447c5-w4rw6\" (UID: \"3ae23680-f7b9-40db-9826-78ec43a1b8b9\") " pod="odh-ai-gateway-infra/maas-api-76cf6447c5-w4rw6"
Sep 18 13:58:19.761521 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.761464    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b89985555-7rwhr"
Sep 18 13:58:19.838226 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.838187    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f97f4c0c-7cdd-43e1-9012-5cf42b610b7e-openshift-service-ca\") pod \"f97f4c0c-7cdd-43e1-9012-5cf42b610b7e\" (UID: \"f97f4c0c-7cdd-43e1-9012-5cf42b610b7e\") "
Sep 18 13:58:19.838421 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.838273    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f97f4c0c-7cdd-43e1-9012-5cf42b610b7e-metrics-certs\") pod \"f97f4c0c-7cdd-43e1-9012-5cf42b610b7e\" (UID: \"f97f4c0c-7cdd-43e1-9012-5cf42b610b7e\") "
Sep 18 13:58:19.838421 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.838300    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f97f4c0c-7cdd-43e1-9012-5cf42b610b7e-maas-api-tls\") pod \"f97f4c0c-7cdd-43e1-9012-5cf42b610b7e\" (UID: \"f97f4c0c-7cdd-43e1-9012-5cf42b610b7e\") "
Sep 18 13:58:19.838421 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.838361    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7w4wz\" (UniqueName: \"kubernetes.io/projected/f97f4c0c-7cdd-43e1-9012-5cf42b610b7e-kube-api-access-7w4wz\") pod \"f97f4c0c-7cdd-43e1-9012-5cf42b610b7e\" (UID: \"f97f4c0c-7cdd-43e1-9012-5cf42b610b7e\") "
Sep 18 13:58:19.838638 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.838612    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f97f4c0c-7cdd-43e1-9012-5cf42b610b7e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f97f4c0c-7cdd-43e1-9012-5cf42b610b7e" (UID: "f97f4c0c-7cdd-43e1-9012-5cf42b610b7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 13:58:19.840742 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.840710    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f97f4c0c-7cdd-43e1-9012-5cf42b610b7e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f97f4c0c-7cdd-43e1-9012-5cf42b610b7e" (UID: "f97f4c0c-7cdd-43e1-9012-5cf42b610b7e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:58:19.840742 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.840733    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97f4c0c-7cdd-43e1-9012-5cf42b610b7e-kube-api-access-7w4wz" (OuterVolumeSpecName: "kube-api-access-7w4wz") pod "f97f4c0c-7cdd-43e1-9012-5cf42b610b7e" (UID: "f97f4c0c-7cdd-43e1-9012-5cf42b610b7e"). InnerVolumeSpecName "kube-api-access-7w4wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 13:58:19.840886 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.840764    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f97f4c0c-7cdd-43e1-9012-5cf42b610b7e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f97f4c0c-7cdd-43e1-9012-5cf42b610b7e" (UID: "f97f4c0c-7cdd-43e1-9012-5cf42b610b7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 13:58:19.894810 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.894758    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76cf6447c5-w4rw6"
Sep 18 13:58:19.939712 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.939655    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w4wz\" (UniqueName: \"kubernetes.io/projected/f97f4c0c-7cdd-43e1-9012-5cf42b610b7e-kube-api-access-7w4wz\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:58:19.939712 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.939716    2586 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f97f4c0c-7cdd-43e1-9012-5cf42b610b7e-openshift-service-ca\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:58:19.939984 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.939730    2586 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f97f4c0c-7cdd-43e1-9012-5cf42b610b7e-metrics-certs\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:58:19.939984 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:19.939744    2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f97f4c0c-7cdd-43e1-9012-5cf42b610b7e-maas-api-tls\") on node \"ip-10-0-131-221.ec2.internal\" DevicePath \"\""
Sep 18 13:58:20.166779 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:20.166672    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76cf6447c5-w4rw6"]
Sep 18 13:58:20.169511 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:58:20.169477    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae23680_f7b9_40db_9826_78ec43a1b8b9.slice/crio-c6ec0a337eabd16b6fd0eff2bb87c8a7a5b77b697b5480bda9be9f070d70ec4d WatchSource:0}: Error finding container c6ec0a337eabd16b6fd0eff2bb87c8a7a5b77b697b5480bda9be9f070d70ec4d: Status 404 returned error can't find the container with id c6ec0a337eabd16b6fd0eff2bb87c8a7a5b77b697b5480bda9be9f070d70ec4d
Sep 18 13:58:20.589817 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:20.589791    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76cf6447c5-w4rw6" event={"ID":"3ae23680-f7b9-40db-9826-78ec43a1b8b9","Type":"ContainerStarted","Data":"c6ec0a337eabd16b6fd0eff2bb87c8a7a5b77b697b5480bda9be9f070d70ec4d"}
Sep 18 13:58:20.591145 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:20.591124    2586 generic.go:356] "Generic (PLEG): container finished" podID="f97f4c0c-7cdd-43e1-9012-5cf42b610b7e" containerID="e08cac5ef8d346cb99cdbda7b707c6f813397936ceaa5a0ab932e8303de8e40a" exitCode=0
Sep 18 13:58:20.591260 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:20.591179    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b89985555-7rwhr"
Sep 18 13:58:20.591260 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:20.591207    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b89985555-7rwhr" event={"ID":"f97f4c0c-7cdd-43e1-9012-5cf42b610b7e","Type":"ContainerDied","Data":"e08cac5ef8d346cb99cdbda7b707c6f813397936ceaa5a0ab932e8303de8e40a"}
Sep 18 13:58:20.591260 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:20.591250    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b89985555-7rwhr" event={"ID":"f97f4c0c-7cdd-43e1-9012-5cf42b610b7e","Type":"ContainerDied","Data":"0b65f4921186a24d3bd3c7d633c4c428bbcbdfdab7b0837bd96180922da44a65"}
Sep 18 13:58:20.591414 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:20.591273    2586 scope.go:117] "RemoveContainer" containerID="e08cac5ef8d346cb99cdbda7b707c6f813397936ceaa5a0ab932e8303de8e40a"
Sep 18 13:58:20.611275 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:20.611254    2586 scope.go:117] "RemoveContainer" containerID="e08cac5ef8d346cb99cdbda7b707c6f813397936ceaa5a0ab932e8303de8e40a"
Sep 18 13:58:20.611571 ip-10-0-131-221 kubenswrapper[2586]: E0918 13:58:20.611555    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e08cac5ef8d346cb99cdbda7b707c6f813397936ceaa5a0ab932e8303de8e40a\": container with ID starting with e08cac5ef8d346cb99cdbda7b707c6f813397936ceaa5a0ab932e8303de8e40a not found: ID does not exist" containerID="e08cac5ef8d346cb99cdbda7b707c6f813397936ceaa5a0ab932e8303de8e40a"
Sep 18 13:58:20.611628 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:20.611580    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e08cac5ef8d346cb99cdbda7b707c6f813397936ceaa5a0ab932e8303de8e40a"} err="failed to get container status \"e08cac5ef8d346cb99cdbda7b707c6f813397936ceaa5a0ab932e8303de8e40a\": rpc error: code = NotFound desc = could not find container \"e08cac5ef8d346cb99cdbda7b707c6f813397936ceaa5a0ab932e8303de8e40a\": container with ID starting with e08cac5ef8d346cb99cdbda7b707c6f813397936ceaa5a0ab932e8303de8e40a not found: ID does not exist"
Sep 18 13:58:20.634516 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:20.634492    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b89985555-7rwhr"]
Sep 18 13:58:20.640124 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:20.640104    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b89985555-7rwhr"]
Sep 18 13:58:20.849583 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:20.849505    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f97f4c0c-7cdd-43e1-9012-5cf42b610b7e" path="/var/lib/kubelet/pods/f97f4c0c-7cdd-43e1-9012-5cf42b610b7e/volumes"
Sep 18 13:58:21.597843 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:21.597802    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76cf6447c5-w4rw6" event={"ID":"3ae23680-f7b9-40db-9826-78ec43a1b8b9","Type":"ContainerStarted","Data":"98b206e248191cc26ee44f14f76b7c9bfe63eed2dbbd1b7d2e00d418347f21a9"}
Sep 18 13:58:21.598328 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:21.597869    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-76cf6447c5-w4rw6"
Sep 18 13:58:21.615152 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:21.615100    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-76cf6447c5-w4rw6" podStartSLOduration=2.202779152 podStartE2EDuration="2.615085029s" totalImagesPullingTime="412.305877ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 13:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:58:21.613561225 +0000 UTC m=+2598.969021049" watchObservedRunningTime="2026-09-18 13:58:21.615085029 +0000 UTC m=+2598.970544854"
Sep 18 13:58:26.592241 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:26.592203    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:58:27.609151 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:27.609117    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-76cf6447c5-w4rw6"
Sep 18 13:58:32.170546 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:32.170510    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:58:38.868609 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:38.868577    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-9stmh"]
Sep 18 13:58:47.528281 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:47.528252    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-ws7zp_3f0b1bf8-da36-4c37-8600-b5a16a85246f/manager/0.log"
Sep 18 13:58:48.156406 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:48.156375    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-854585bc8d-g56vw_52171301-42af-4d64-a163-908f491dd2e1/copy-manifests/0.log"
Sep 18 13:58:48.168652 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:48.168621    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-854585bc8d-g56vw_52171301-42af-4d64-a163-908f491dd2e1/manager/0.log"
Sep 18 13:58:48.760735 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:48.760700    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-76cf6447c5-w4rw6_3ae23680-f7b9-40db-9826-78ec43a1b8b9/maas-api/0.log"
Sep 18 13:58:49.003086 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:49.003050    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29828985-9fvm9_e2424bf3-4834-4884-8c25-9f7f924769c9/cleanup/0.log"
Sep 18 13:58:50.421733 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:50.421695    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-q4m2d_7f0e9d3d-14b8-4a2d-a634-a2ee26719f68/manager/0.log"
Sep 18 13:58:50.539926 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:50.539870    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-9stmh_70be07c6-1642-4f67-baea-3a49b4006e07/limitador/0.log"
Sep 18 13:58:52.352644 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:52.352612    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-6pxr8_a9414bb6-2317-4e59-bbd8-8ebc5096749c/main/0.log"
Sep 18 13:58:52.466112 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:52.466088    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-vkcfv_755a9ecb-c5f9-4c8a-82b8-eaef5989ab70/main/0.log"
Sep 18 13:58:52.692950 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:52.692921    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fm6v6q_c9b485d0-4b37-4766-8d31-02c8fda334c2/main/0.log"
Sep 18 13:58:52.804943 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:58:52.804889    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-7z2lm_3676d2ca-51a1-4436-967e-af28a0e41be1/main/0.log"
Sep 18 13:59:01.598011 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:01.597973    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pc5zc_7e310da4-ab19-4631-9717-86fb07925256/global-pull-secret-syncer/0.log"
Sep 18 13:59:01.747775 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:01.747745    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fmxrm_e53a6d50-eb67-499c-9626-d7788ef58cef/konnectivity-agent/0.log"
Sep 18 13:59:01.803614 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:01.803580    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-221.ec2.internal_523614729ae9b688582b641ea2fafde6/haproxy/0.log"
Sep 18 13:59:07.051948 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:07.051894    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-q4m2d_7f0e9d3d-14b8-4a2d-a634-a2ee26719f68/manager/0.log"
Sep 18 13:59:07.074754 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:07.074723    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-9stmh_70be07c6-1642-4f67-baea-3a49b4006e07/limitador/0.log"
Sep 18 13:59:09.035068 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:09.035040    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-76bb975779-mgnrc_c7cd73e0-5d9c-46be-9532-5661607b7b39/metrics-server/0.log"
Sep 18 13:59:09.120269 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:09.120199    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b6t2b_9e5f1d5b-d9ca-4a0b-985b-4b0147312765/node-exporter/0.log"
Sep 18 13:59:09.143115 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:09.143094    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b6t2b_9e5f1d5b-d9ca-4a0b-985b-4b0147312765/kube-rbac-proxy/0.log"
Sep 18 13:59:09.166525 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:09.166506    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b6t2b_9e5f1d5b-d9ca-4a0b-985b-4b0147312765/init-textfile/0.log"
Sep 18 13:59:09.478346 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:09.478304    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_80708264-fa72-4841-9597-3c902b788ac1/prometheus/0.log"
Sep 18 13:59:09.502326 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:09.502305    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_80708264-fa72-4841-9597-3c902b788ac1/config-reloader/0.log"
Sep 18 13:59:09.525507 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:09.525455    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_80708264-fa72-4841-9597-3c902b788ac1/thanos-sidecar/0.log"
Sep 18 13:59:09.546629 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:09.546611    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_80708264-fa72-4841-9597-3c902b788ac1/kube-rbac-proxy-web/0.log"
Sep 18 13:59:09.568400 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:09.568375    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_80708264-fa72-4841-9597-3c902b788ac1/kube-rbac-proxy/0.log"
Sep 18 13:59:09.593612 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:09.593590    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_80708264-fa72-4841-9597-3c902b788ac1/kube-rbac-proxy-thanos/0.log"
Sep 18 13:59:09.612980 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:09.612955    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_80708264-fa72-4841-9597-3c902b788ac1/init-config-reloader/0.log"
Sep 18 13:59:11.311239 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.311207    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gnq97/perf-node-gather-daemonset-6nnjf"]
Sep 18 13:59:11.311692 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.311539    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f97f4c0c-7cdd-43e1-9012-5cf42b610b7e" containerName="maas-api"
Sep 18 13:59:11.311692 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.311554    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f97f4c0c-7cdd-43e1-9012-5cf42b610b7e" containerName="maas-api"
Sep 18 13:59:11.311692 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.311616    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f97f4c0c-7cdd-43e1-9012-5cf42b610b7e" containerName="maas-api"
Sep 18 13:59:11.316042 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.316025    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-6nnjf"
Sep 18 13:59:11.317974 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.317954    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gnq97\"/\"openshift-service-ca.crt\""
Sep 18 13:59:11.318306 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.318287    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gnq97\"/\"default-dockercfg-jvgfk\""
Sep 18 13:59:11.318306 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.318302    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gnq97\"/\"kube-root-ca.crt\""
Sep 18 13:59:11.323315 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.323294    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gnq97/perf-node-gather-daemonset-6nnjf"]
Sep 18 13:59:11.393137 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.393105    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/65c95edc-0083-4a3f-be41-e246e568fb4d-podres\") pod \"perf-node-gather-daemonset-6nnjf\" (UID: \"65c95edc-0083-4a3f-be41-e246e568fb4d\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-6nnjf"
Sep 18 13:59:11.393137 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.393135    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/65c95edc-0083-4a3f-be41-e246e568fb4d-proc\") pod \"perf-node-gather-daemonset-6nnjf\" (UID: \"65c95edc-0083-4a3f-be41-e246e568fb4d\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-6nnjf"
Sep 18 13:59:11.393336 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.393243    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwz85\" (UniqueName: \"kubernetes.io/projected/65c95edc-0083-4a3f-be41-e246e568fb4d-kube-api-access-nwz85\") pod \"perf-node-gather-daemonset-6nnjf\" (UID: \"65c95edc-0083-4a3f-be41-e246e568fb4d\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-6nnjf"
Sep 18 13:59:11.393336 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.393289    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65c95edc-0083-4a3f-be41-e246e568fb4d-lib-modules\") pod \"perf-node-gather-daemonset-6nnjf\" (UID: \"65c95edc-0083-4a3f-be41-e246e568fb4d\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-6nnjf"
Sep 18 13:59:11.393336 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.393313    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65c95edc-0083-4a3f-be41-e246e568fb4d-sys\") pod \"perf-node-gather-daemonset-6nnjf\" (UID: \"65c95edc-0083-4a3f-be41-e246e568fb4d\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-6nnjf"
Sep 18 13:59:11.493922 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.493876    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwz85\" (UniqueName: \"kubernetes.io/projected/65c95edc-0083-4a3f-be41-e246e568fb4d-kube-api-access-nwz85\") pod \"perf-node-gather-daemonset-6nnjf\" (UID: \"65c95edc-0083-4a3f-be41-e246e568fb4d\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-6nnjf"
Sep 18 13:59:11.494127 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.493966    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65c95edc-0083-4a3f-be41-e246e568fb4d-lib-modules\") pod \"perf-node-gather-daemonset-6nnjf\" (UID: \"65c95edc-0083-4a3f-be41-e246e568fb4d\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-6nnjf"
Sep 18 13:59:11.494127 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.494097    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65c95edc-0083-4a3f-be41-e246e568fb4d-lib-modules\") pod \"perf-node-gather-daemonset-6nnjf\" (UID: \"65c95edc-0083-4a3f-be41-e246e568fb4d\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-6nnjf"
Sep 18 13:59:11.494127 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.494099    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65c95edc-0083-4a3f-be41-e246e568fb4d-sys\") pod \"perf-node-gather-daemonset-6nnjf\" (UID: \"65c95edc-0083-4a3f-be41-e246e568fb4d\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-6nnjf"
Sep 18 13:59:11.494286 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.494147    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/65c95edc-0083-4a3f-be41-e246e568fb4d-podres\") pod \"perf-node-gather-daemonset-6nnjf\" (UID: \"65c95edc-0083-4a3f-be41-e246e568fb4d\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-6nnjf"
Sep 18 13:59:11.494286 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.494168    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/65c95edc-0083-4a3f-be41-e246e568fb4d-proc\") pod \"perf-node-gather-daemonset-6nnjf\" (UID: \"65c95edc-0083-4a3f-be41-e246e568fb4d\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-6nnjf"
Sep 18 13:59:11.494286 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.494146    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65c95edc-0083-4a3f-be41-e246e568fb4d-sys\") pod \"perf-node-gather-daemonset-6nnjf\" (UID: \"65c95edc-0083-4a3f-be41-e246e568fb4d\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-6nnjf"
Sep 18 13:59:11.494286 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.494241    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/65c95edc-0083-4a3f-be41-e246e568fb4d-podres\") pod \"perf-node-gather-daemonset-6nnjf\" (UID: \"65c95edc-0083-4a3f-be41-e246e568fb4d\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-6nnjf"
Sep 18 13:59:11.494421 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.494297    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/65c95edc-0083-4a3f-be41-e246e568fb4d-proc\") pod \"perf-node-gather-daemonset-6nnjf\" (UID: \"65c95edc-0083-4a3f-be41-e246e568fb4d\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-6nnjf"
Sep 18 13:59:11.501721 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.501695    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwz85\" (UniqueName: \"kubernetes.io/projected/65c95edc-0083-4a3f-be41-e246e568fb4d-kube-api-access-nwz85\") pod \"perf-node-gather-daemonset-6nnjf\" (UID: \"65c95edc-0083-4a3f-be41-e246e568fb4d\") " pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-6nnjf"
Sep 18 13:59:11.625705 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.625627    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-6nnjf"
Sep 18 13:59:11.900467 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:11.900432    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gnq97/perf-node-gather-daemonset-6nnjf"]
Sep 18 13:59:11.904707 ip-10-0-131-221 kubenswrapper[2586]: W0918 13:59:11.904670    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod65c95edc_0083_4a3f_be41_e246e568fb4d.slice/crio-e5f5f82c889995f24980f6fd2907be6526ed3cd8f21c99a5ae311f30ce79e5d5 WatchSource:0}: Error finding container e5f5f82c889995f24980f6fd2907be6526ed3cd8f21c99a5ae311f30ce79e5d5: Status 404 returned error can't find the container with id e5f5f82c889995f24980f6fd2907be6526ed3cd8f21c99a5ae311f30ce79e5d5
Sep 18 13:59:12.826707 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:12.826666    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-6nnjf" event={"ID":"65c95edc-0083-4a3f-be41-e246e568fb4d","Type":"ContainerStarted","Data":"7d7824121252ef8752838c1dbf899e3149e15d853cf24e930d6e9aca2e5df3ba"}
Sep 18 13:59:12.827239 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:12.826759    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-6nnjf" event={"ID":"65c95edc-0083-4a3f-be41-e246e568fb4d","Type":"ContainerStarted","Data":"e5f5f82c889995f24980f6fd2907be6526ed3cd8f21c99a5ae311f30ce79e5d5"}
Sep 18 13:59:12.827239 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:12.826790    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-6nnjf"
Sep 18 13:59:12.842800 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:12.842729    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-6nnjf" podStartSLOduration=1.842714531 podStartE2EDuration="1.842714531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 13:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 13:59:12.841615963 +0000 UTC m=+2650.197075780" watchObservedRunningTime="2026-09-18 13:59:12.842714531 +0000 UTC m=+2650.198174367"
Sep 18 13:59:14.359552 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:14.359519    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hp46h_617fc06c-1fc8-479a-badc-dfa07662e6b0/dns/0.log"
Sep 18 13:59:14.395738 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:14.395710    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hp46h_617fc06c-1fc8-479a-badc-dfa07662e6b0/kube-rbac-proxy/0.log"
Sep 18 13:59:14.646770 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:14.646739    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lbfgj_73749cf1-5bc4-452a-b1a4-b3ef009e3ff3/dns-node-resolver/0.log"
Sep 18 13:59:15.266793 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:15.266759    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6645f89c6b-svklk_a7fe2b0a-4e0d-483b-a53a-cd6387f695cb/registry/0.log"
Sep 18 13:59:15.341525 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:15.341495    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s99br_7721f73b-5184-4404-b65b-774dfff4df76/node-ca/0.log"
Sep 18 13:59:18.017716 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:18.017684    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-crvdj_186f3f17-3593-4843-985c-b4bb77d3f28c/serve-healthcheck-canary/0.log"
Sep 18 13:59:18.669954 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:18.669921    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g5mh6_d0e99754-f349-4a91-8b6e-5f9622418657/kube-rbac-proxy/0.log"
Sep 18 13:59:18.712362 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:18.712334    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g5mh6_d0e99754-f349-4a91-8b6e-5f9622418657/exporter/0.log"
Sep 18 13:59:18.760449 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:18.760425    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-g5mh6_d0e99754-f349-4a91-8b6e-5f9622418657/extractor/0.log"
Sep 18 13:59:18.841173 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:18.841147    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gnq97/perf-node-gather-daemonset-6nnjf"
Sep 18 13:59:21.757243 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:21.757148    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-ws7zp_3f0b1bf8-da36-4c37-8600-b5a16a85246f/manager/0.log"
Sep 18 13:59:22.100570 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:22.100496    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-854585bc8d-g56vw_52171301-42af-4d64-a163-908f491dd2e1/manager/0.log"
Sep 18 13:59:22.120828 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:22.120803    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-854585bc8d-g56vw_52171301-42af-4d64-a163-908f491dd2e1/copy-manifests/0.log"
Sep 18 13:59:23.574548 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:23.574517    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-85985676b5-vqt6t_4846c6d7-ea37-4392-9c57-ec4b348b5684/manager/0.log"
Sep 18 13:59:28.849419 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:28.849392    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-9q8d4_ea66c1bc-2ed4-436e-ade9-3545d118cd80/migrator/0.log"
Sep 18 13:59:28.873201 ip-10-0-131-221 kubenswrapper[2586]: I0918 13:59:28.873177    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-9q8d4_ea66c1bc-2ed4-436e-ade9-3545d118cd80/graceful-termination/0.log"