Sep 11 21:50:41.141214 ip-10-0-139-88 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 21:50:41.141224 ip-10-0-139-88 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 21:50:41.141231 ip-10-0-139-88 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 21:50:41.141456 ip-10-0-139-88 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 21:50:52.813399 ip-10-0-139-88 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 21:50:52.813415 ip-10-0-139-88 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 99f96c001d5c43d38d35ca914f86b543 --
Sep 11 21:52:25.304833 ip-10-0-139-88 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 21:52:25.736912 ip-10-0-139-88 kubenswrapper[2587]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 21:52:25.736912 ip-10-0-139-88 kubenswrapper[2587]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 21:52:25.736912 ip-10-0-139-88 kubenswrapper[2587]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 21:52:25.736912 ip-10-0-139-88 kubenswrapper[2587]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 21:52:25.736912 ip-10-0-139-88 kubenswrapper[2587]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 21:52:25.737879 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.737792    2587 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 21:52:25.741336 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741320    2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:52:25.741336 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741336    2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:52:25.741400 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741341    2587 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:52:25.741400 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741345    2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:52:25.741400 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741348    2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:52:25.741400 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741351    2587 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:52:25.741400 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741354    2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:52:25.741400 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741357    2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:52:25.741400 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741360    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:52:25.741400 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741362    2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:52:25.741400 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741365    2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:52:25.741400 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741368    2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:52:25.741400 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741371    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:52:25.741400 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741374    2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:52:25.741400 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741376    2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:52:25.741400 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741379    2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:52:25.741400 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741381    2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:52:25.741400 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741390    2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:52:25.741400 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741393    2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:52:25.741400 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741395    2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:52:25.741400 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741398    2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:52:25.741400 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741400    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:52:25.741894 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741404    2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:52:25.741894 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741408    2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:52:25.741894 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741411    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:52:25.741894 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741414    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:52:25.741894 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741417    2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:52:25.741894 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741424    2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:52:25.741894 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741427    2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:52:25.741894 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741430    2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:52:25.741894 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741432    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:52:25.741894 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741435    2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:52:25.741894 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741437    2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:52:25.741894 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741440    2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:52:25.741894 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741442    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:52:25.741894 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741445    2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:52:25.741894 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741448    2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:52:25.741894 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741450    2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:52:25.741894 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741453    2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:52:25.741894 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741455    2587 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:52:25.741894 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741458    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:52:25.741894 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741460    2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:52:25.742382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741463    2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:52:25.742382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741465    2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:52:25.742382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741468    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:52:25.742382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741470    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:52:25.742382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741473    2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:52:25.742382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741475    2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:52:25.742382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741478    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:52:25.742382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741480    2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:52:25.742382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741483    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:52:25.742382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741485    2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:52:25.742382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741488    2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:52:25.742382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741492    2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:52:25.742382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741509    2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:52:25.742382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741513    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:52:25.742382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741517    2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:52:25.742382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741520    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:52:25.742382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741522    2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:52:25.742382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741525    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:52:25.742382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741528    2587 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:52:25.742852 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741530    2587 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:52:25.742852 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741533    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:52:25.742852 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741536    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:52:25.742852 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741540    2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:52:25.742852 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741543    2587 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:52:25.742852 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741546    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:52:25.742852 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741549    2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:52:25.742852 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741552    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:52:25.742852 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741555    2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:52:25.742852 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741558    2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:52:25.742852 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741560    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:52:25.742852 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741563    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:52:25.742852 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741565    2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:52:25.742852 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741568    2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:52:25.742852 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741570    2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:52:25.742852 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741573    2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:52:25.742852 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741575    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:52:25.742852 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741579    2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:52:25.742852 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741583    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:52:25.743391 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741586    2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:52:25.743391 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741588    2587 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:52:25.743391 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741591    2587 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:52:25.743391 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741593    2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:52:25.743391 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741596    2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:52:25.743391 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741599    2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:52:25.743391 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741602    2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:52:25.743391 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741606    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:52:25.743391 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741609    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:52:25.743391 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741612    2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:52:25.743391 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741614    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:52:25.743391 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741617    2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:52:25.743391 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741620    2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:52:25.743391 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741622    2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:52:25.743391 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741625    2587 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:52:25.743391 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741628    2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:52:25.743391 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741630    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:52:25.743391 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741633    2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:52:25.743391 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.741636    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:52:25.743391 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742325    2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:52:25.743941 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742331    2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:52:25.743941 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742335    2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:52:25.743941 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742338    2587 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:52:25.743941 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742341    2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:52:25.743941 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742344    2587 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:52:25.743941 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742346    2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:52:25.743941 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742350    2587 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:52:25.743941 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742352    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:52:25.743941 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742355    2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:52:25.743941 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742358    2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:52:25.743941 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742360    2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:52:25.743941 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742363    2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:52:25.743941 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742366    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:52:25.743941 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742370    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:52:25.743941 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742373    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:52:25.743941 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742375    2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:52:25.743941 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742378    2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:52:25.743941 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742381    2587 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:52:25.743941 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742383    2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:52:25.743941 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742387    2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:52:25.744437 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742390    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:52:25.744437 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742393    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:52:25.744437 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742396    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:52:25.744437 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742399    2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:52:25.744437 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742401    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:52:25.744437 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742404    2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:52:25.744437 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742408    2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:52:25.744437 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742412    2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:52:25.744437 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742415    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:52:25.744437 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742418    2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:52:25.744437 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742421    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:52:25.744437 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742423    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:52:25.744437 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742426    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:52:25.744437 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742429    2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:52:25.744437 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742431    2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:52:25.744437 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742434    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:52:25.744437 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742437    2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:52:25.744437 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742439    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:52:25.744437 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742442    2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:52:25.744927 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742445    2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:52:25.744927 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742447    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:52:25.744927 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742450    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:52:25.744927 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742454    2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:52:25.744927 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742457    2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:52:25.744927 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742460    2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:52:25.744927 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742463    2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:52:25.744927 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742466    2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:52:25.744927 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742468    2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:52:25.744927 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742471    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:52:25.744927 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742474    2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:52:25.744927 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742476    2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:52:25.744927 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742479    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:52:25.744927 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742482    2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:52:25.744927 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742485    2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:52:25.744927 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742488    2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:52:25.744927 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742491    2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:52:25.744927 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742508    2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:52:25.744927 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742511    2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:52:25.744927 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742514    2587 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:52:25.745413 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742517    2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:52:25.745413 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742519    2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:52:25.745413 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742522    2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:52:25.745413 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742526    2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:52:25.745413 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742529    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:52:25.745413 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742532    2587 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:52:25.745413 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742535    2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:52:25.745413 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742538    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:52:25.745413 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742541    2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:52:25.745413 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742544    2587 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:52:25.745413 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742547    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:52:25.745413 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742550    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:52:25.745413 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742552    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:52:25.745413 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742555    2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:52:25.745413 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742558    2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:52:25.745413 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742561    2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:52:25.745413 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742563    2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:52:25.745413 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742566    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:52:25.745413 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742568    2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:52:25.745898 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742578    2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:52:25.745898 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742581    2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:52:25.745898 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742583    2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:52:25.745898 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742586    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:52:25.745898 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742588    2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:52:25.745898 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742592    2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:52:25.745898 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742594    2587 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:52:25.745898 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742597    2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:52:25.745898 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742600    2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:52:25.745898 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742603    2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:52:25.745898 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742606    2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:52:25.745898 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742608    2587 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:52:25.745898 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742611    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:52:25.745898 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742614    2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:52:25.745898 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742616    2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:52:25.745898 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742619    2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:52:25.745898 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742622    2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:52:25.745898 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742625    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:52:25.745898 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742628    2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:52:25.745898 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742631    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:52:25.746382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742992    2587 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:52:25.746382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742996    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:52:25.746382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.742999    2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:52:25.746382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743002    2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:52:25.746382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743005    2587 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:52:25.746382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743007    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:52:25.746382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743010    2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:52:25.746382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743013    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:52:25.746382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743016    2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:52:25.746382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743018    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:52:25.746382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743021    2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:52:25.746382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743024    2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:52:25.746382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743027    2587 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:52:25.746382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743029    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:52:25.746382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743032    2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:52:25.746382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743035    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:52:25.746382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743037    2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:52:25.746382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743042    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:52:25.746382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743046    2587 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:52:25.746382 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743048    2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:52:25.746874 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743053    2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:52:25.746874 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743056    2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:52:25.746874 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743060    2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:52:25.746874 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743063    2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:52:25.746874 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743065    2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:52:25.746874 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743068    2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:52:25.746874 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743071    2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:52:25.746874 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743073    2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:52:25.746874 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743075    2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:52:25.746874 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743078    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:52:25.746874 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743080    2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:52:25.746874 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743083    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:52:25.746874 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743086    2587 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:52:25.746874 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743089    2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:52:25.746874 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743092    2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:52:25.746874 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743094    2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:52:25.746874 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743097    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:52:25.746874 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743099    2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:52:25.746874 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743102    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:52:25.747325 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743104    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:52:25.747325 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743107    2587 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:52:25.747325 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743110    2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:52:25.747325 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743112    2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:52:25.747325 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743115    2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:52:25.747325 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743117    2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:52:25.747325 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743121    2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:52:25.747325 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743123    2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:52:25.747325 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743126    2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:52:25.747325 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743129    2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:52:25.747325 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743133    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:52:25.747325 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743136    2587 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:52:25.747325 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743138    2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:52:25.747325 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743142    2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:52:25.747325 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743146    2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:52:25.747325 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743149    2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:52:25.747325 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743152    2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:52:25.747325 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743154    2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:52:25.747325 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743157    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:52:25.747814 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743160    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:52:25.747814 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743162    2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:52:25.747814 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743166    2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:52:25.747814 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743170    2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:52:25.747814 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743172    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:52:25.747814 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743175    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:52:25.747814 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743178    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:52:25.747814 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743180    2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:52:25.747814 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743183    2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:52:25.747814 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743186    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:52:25.747814 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743188    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:52:25.747814 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743191    2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:52:25.747814 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743193    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:52:25.747814 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743197    2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:52:25.747814 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743200    2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:52:25.747814 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743202    2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:52:25.747814 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743205    2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:52:25.747814 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743207    2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:52:25.747814 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743210    2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:52:25.748326 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743212    2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:52:25.748326 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743215    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:52:25.748326 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743218    2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:52:25.748326 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743221    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:52:25.748326 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743226    2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:52:25.748326 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743228    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:52:25.748326 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743231    2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:52:25.748326 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743233    2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:52:25.748326 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743236    2587 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:52:25.748326 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743239    2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:52:25.748326 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743241    2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:52:25.748326 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743244    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:52:25.748326 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743246    2587 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:52:25.748326 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743249    2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:52:25.748326 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743251    2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:52:25.748326 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743255    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:52:25.748326 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743258    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:52:25.748326 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743260    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:52:25.748326 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743263    2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:52:25.748326 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743265    2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:52:25.748825 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743268    2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:52:25.748825 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.743271    2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:52:25.748825 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743349    2587 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 21:52:25.748825 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743384    2587 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 21:52:25.748825 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743392    2587 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 21:52:25.748825 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743397    2587 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 21:52:25.748825 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743401    2587 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 21:52:25.748825 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743405    2587 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 21:52:25.748825 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743409    2587 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 21:52:25.748825 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743413    2587 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 21:52:25.748825 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743416    2587 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 21:52:25.748825 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743420    2587 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 21:52:25.748825 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743423    2587 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 21:52:25.748825 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743426    2587 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 21:52:25.748825 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743430    2587 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 21:52:25.748825 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743433    2587 flags.go:64] FLAG: --cgroup-root=""
Sep 11 21:52:25.748825 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743437    2587 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 21:52:25.748825 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743440    2587 flags.go:64] FLAG: --client-ca-file=""
Sep 11 21:52:25.748825 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743442    2587 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 21:52:25.748825 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743445    2587 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 21:52:25.748825 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743450    2587 flags.go:64] FLAG: --cluster-domain=""
Sep 11 21:52:25.748825 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743452    2587 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 21:52:25.748825 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743456    2587 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 21:52:25.749374 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743459    2587 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 21:52:25.749374 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743462    2587 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 21:52:25.749374 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743466    2587 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 21:52:25.749374 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743469    2587 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 21:52:25.749374 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743473    2587 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 21:52:25.749374 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743476    2587 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 21:52:25.749374 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743479    2587 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 21:52:25.749374 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743482    2587 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 21:52:25.749374 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743485    2587 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 21:52:25.749374 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743488    2587 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 21:52:25.749374 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743491    2587 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 21:52:25.749374 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743512    2587 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 21:52:25.749374 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743516    2587 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 21:52:25.749374 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743521    2587 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 21:52:25.749374 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743525    2587 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 21:52:25.749374 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743530    2587 flags.go:64] FLAG: --enable-server="true"
Sep 11 21:52:25.749374 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743533    2587 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 21:52:25.749374 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743538    2587 flags.go:64] FLAG: --event-burst="100"
Sep 11 21:52:25.749374 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743541    2587 flags.go:64] FLAG: --event-qps="50"
Sep 11 21:52:25.749374 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743544    2587 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 21:52:25.749374 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743547    2587 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 21:52:25.749374 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743550    2587 flags.go:64] FLAG: --eviction-hard=""
Sep 11 21:52:25.749374 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743554    2587 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 21:52:25.749374 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743556    2587 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 21:52:25.749959 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743560    2587 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 21:52:25.749959 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743563    2587 flags.go:64] FLAG: --eviction-soft=""
Sep 11 21:52:25.749959 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743567    2587 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 21:52:25.749959 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743570    2587 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 21:52:25.749959 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743573    2587 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 21:52:25.749959 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743576    2587 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 21:52:25.749959 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743579    2587 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 21:52:25.749959 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743582    2587 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 21:52:25.749959 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743585    2587 flags.go:64] FLAG: --feature-gates=""
Sep 11 21:52:25.749959 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743589    2587 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 21:52:25.749959 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743592    2587 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 21:52:25.749959 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743595    2587 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 21:52:25.749959 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743599    2587 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 21:52:25.749959 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743602    2587 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 21:52:25.749959 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743605    2587 flags.go:64] FLAG: --help="false"
Sep 11 21:52:25.749959 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743608    2587 flags.go:64] FLAG: --hostname-override="ip-10-0-139-88.ec2.internal"
Sep 11 21:52:25.749959 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743611    2587 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 21:52:25.749959 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743615    2587 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 21:52:25.749959 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743618    2587 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 21:52:25.749959 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743621    2587 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 21:52:25.749959 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743624    2587 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 21:52:25.749959 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743627    2587 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 21:52:25.749959 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743630    2587 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 21:52:25.750563 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743633    2587 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 21:52:25.750563 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743636    2587 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 21:52:25.750563 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743639    2587 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 21:52:25.750563 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743642    2587 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 21:52:25.750563 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743645    2587 flags.go:64] FLAG: --kube-reserved=""
Sep 11 21:52:25.750563 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743648    2587 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 21:52:25.750563 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743651    2587 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 21:52:25.750563 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743654    2587 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 21:52:25.750563 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743657    2587 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 21:52:25.750563 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743666    2587 flags.go:64] FLAG: --lock-file=""
Sep 11 21:52:25.750563 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743671    2587 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 21:52:25.750563 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743674    2587 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 21:52:25.750563 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743677    2587 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 21:52:25.750563 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743683    2587 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 21:52:25.750563 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743686    2587 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 21:52:25.750563 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743688    2587 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 21:52:25.750563 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743691    2587 flags.go:64] FLAG: --logging-format="text"
Sep 11 21:52:25.750563 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743694    2587 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 21:52:25.750563 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743697    2587 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 21:52:25.750563 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743700    2587 flags.go:64] FLAG: --manifest-url=""
Sep 11 21:52:25.750563 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743703    2587 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 21:52:25.750563 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743708    2587 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 21:52:25.750563 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743711    2587 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 21:52:25.750563 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743715    2587 flags.go:64] FLAG: --max-pods="110"
Sep 11 21:52:25.750563 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743718    2587 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 21:52:25.751183 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743721    2587 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 21:52:25.751183 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743723    2587 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 21:52:25.751183 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743726    2587 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 21:52:25.751183 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743730    2587 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 21:52:25.751183 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743733    2587 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 21:52:25.751183 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743736    2587 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 21:52:25.751183 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743743    2587 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 21:52:25.751183 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743747    2587 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 21:52:25.751183 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743750    2587 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 21:52:25.751183 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743753    2587 flags.go:64] FLAG: --pod-cidr=""
Sep 11 21:52:25.751183 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743756    2587 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 21:52:25.751183 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743762    2587 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 21:52:25.751183 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743765    2587 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 21:52:25.751183 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743768    2587 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 21:52:25.751183 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743771    2587 flags.go:64] FLAG: --port="10250"
Sep 11 21:52:25.751183 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743774    2587 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 21:52:25.751183 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743777    2587 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0685f9cf51b92c54b"
Sep 11 21:52:25.751183 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743780    2587 flags.go:64] FLAG: --qos-reserved=""
Sep 11 21:52:25.751183 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743786    2587 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 21:52:25.751183 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743790    2587 flags.go:64] FLAG: --register-node="true"
Sep 11 21:52:25.751183 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743793    2587 flags.go:64] FLAG: --register-with-taints=""
Sep 11 21:52:25.751183 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743796    2587 flags.go:64] FLAG: --registry-burst="10"
Sep 11 21:52:25.751183 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743799    2587 flags.go:64] FLAG: --registry-qps="5"
Sep 11 21:52:25.751183 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743802    2587 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 21:52:25.751846 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743805    2587 flags.go:64] FLAG: --reserved-memory=""
Sep 11 21:52:25.751846 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743808    2587 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 21:52:25.751846 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743811    2587 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 21:52:25.751846 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743814    2587 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 21:52:25.751846 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743817    2587 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 21:52:25.751846 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743820    2587 flags.go:64] FLAG: --runonce="false"
Sep 11 21:52:25.751846 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743823    2587 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 21:52:25.751846 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743826    2587 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 21:52:25.751846 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743829    2587 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 21:52:25.751846 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743832    2587 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 21:52:25.751846 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743835    2587 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 21:52:25.751846 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743838    2587 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 21:52:25.751846 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743841    2587 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 21:52:25.751846 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743844    2587 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 21:52:25.751846 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743846    2587 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 21:52:25.751846 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743849    2587 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 21:52:25.751846 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743852    2587 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 21:52:25.751846 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743855    2587 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 21:52:25.751846 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743858    2587 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 21:52:25.751846 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743861    2587 flags.go:64] FLAG: --system-cgroups=""
Sep 11 21:52:25.751846 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743864    2587 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 21:52:25.751846 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743870    2587 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 21:52:25.751846 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743873    2587 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 21:52:25.751846 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743875    2587 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 21:52:25.751846 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743879    2587 flags.go:64] FLAG: --tls-min-version=""
Sep 11 21:52:25.752449 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743882    2587 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 21:52:25.752449 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743886    2587 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 21:52:25.752449 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743890    2587 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 21:52:25.752449 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743892    2587 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 21:52:25.752449 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743895    2587 flags.go:64] FLAG: --v="2"
Sep 11 21:52:25.752449 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743899    2587 flags.go:64] FLAG: --version="false"
Sep 11 21:52:25.752449 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743903    2587 flags.go:64] FLAG: --vmodule=""
Sep 11 21:52:25.752449 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743907    2587 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 21:52:25.752449 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.743910    2587 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 21:52:25.752449 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744026    2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:52:25.752449 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744029    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:52:25.752449 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744034    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:52:25.752449 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744037    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:52:25.752449 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744041    2587 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:52:25.752449 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744044    2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:52:25.752449 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744047    2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:52:25.752449 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744050    2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:52:25.752449 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744052    2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:52:25.752449 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744055    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:52:25.752449 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744058    2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:52:25.752449 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744061    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:52:25.752449 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744063    2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:52:25.753003 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744066    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:52:25.753003 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744069    2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:52:25.753003 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744071    2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:52:25.753003 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744074    2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:52:25.753003 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744079    2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:52:25.753003 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744082    2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:52:25.753003 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744085    2587 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:52:25.753003 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744088    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:52:25.753003 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744092    2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:52:25.753003 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744095    2587 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:52:25.753003 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744098    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:52:25.753003 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744102    2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:52:25.753003 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744105    2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:52:25.753003 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744108    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:52:25.753003 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744111    2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:52:25.753003 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744114    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:52:25.753003 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744117    2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:52:25.753003 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744119    2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:52:25.753003 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744122    2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:52:25.753475 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744125    2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:52:25.753475 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744127    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:52:25.753475 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744131    2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:52:25.753475 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744134    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:52:25.753475 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744137    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:52:25.753475 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744139    2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:52:25.753475 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744142    2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:52:25.753475 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744146    2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:52:25.753475 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744149    2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:52:25.753475 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744152    2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:52:25.753475 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744155    2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:52:25.753475 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744158    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:52:25.753475 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744161    2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:52:25.753475 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744164    2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:52:25.753475 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744167    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:52:25.753475 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744170    2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:52:25.753475 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744173    2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:52:25.753475 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744176    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:52:25.753475 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744178    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:52:25.753475 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744181    2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:52:25.753977 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744184    2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:52:25.753977 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744186    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:52:25.753977 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744189    2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:52:25.753977 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744192    2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:52:25.753977 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744197    2587 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:52:25.753977 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744201    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:52:25.753977 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744204    2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:52:25.753977 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744207    2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:52:25.753977 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744210    2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:52:25.753977 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744213    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:52:25.753977 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744215    2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:52:25.753977 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744218    2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:52:25.753977 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744221    2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:52:25.753977 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744224    2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:52:25.753977 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744228    2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:52:25.753977 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744231    2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:52:25.753977 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744233    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:52:25.753977 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744236    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:52:25.753977 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744238    2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:52:25.754441 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744241    2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:52:25.754441 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744244    2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:52:25.754441 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744246    2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:52:25.754441 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744249    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:52:25.754441 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744251    2587 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:52:25.754441 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744254    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:52:25.754441 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744257    2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:52:25.754441 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744259    2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:52:25.754441 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744262    2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:52:25.754441 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744265    2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:52:25.754441 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744268    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:52:25.754441 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744270    2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:52:25.754441 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744273    2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:52:25.754441 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744276    2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:52:25.754441 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744278    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:52:25.754441 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744281    2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:52:25.754441 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744283    2587 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:52:25.754441 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744287    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:52:25.754441 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744290    2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:52:25.754441 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744293    2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:52:25.754946 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744295    2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:52:25.754946 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744299    2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:52:25.754946 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744302    2587 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:52:25.754946 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744304    2587 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:52:25.754946 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744307    2587 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:52:25.754946 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744310    2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:52:25.754946 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744313    2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:52:25.754946 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.744317    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:52:25.754946 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.745232    2587 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 21:52:25.754946 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.752986    2587 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 21:52:25.754946 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.753001    2587 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 21:52:25.754946 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.753017    2587 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 21:52:25.754946 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.753027    2587 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 21:52:25.754946 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753084    2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:52:25.754946 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753091    2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:52:25.755321 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753095    2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:52:25.755321 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753098    2587 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:52:25.755321 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753101    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:52:25.755321 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753104    2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:52:25.755321 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753107    2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:52:25.755321 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753110    2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:52:25.755321 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753113    2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:52:25.755321 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753116    2587 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:52:25.755321 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753118    2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:52:25.755321 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753121    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:52:25.755321 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753124    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:52:25.755321 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753128    2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:52:25.755321 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753131    2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:52:25.755321 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753134    2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:52:25.755321 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753136    2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:52:25.755321 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753139    2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:52:25.755321 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753142    2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:52:25.755321 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753145    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:52:25.755321 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753148    2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:52:25.755321 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753151    2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:52:25.755839 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753154    2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:52:25.755839 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753156    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:52:25.755839 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753159    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:52:25.755839 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753162    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:52:25.755839 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753165    2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:52:25.755839 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753168    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:52:25.755839 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753170    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:52:25.755839 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753173    2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:52:25.755839 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753176    2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:52:25.755839 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753179    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:52:25.755839 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753182    2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:52:25.755839 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753184    2587 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:52:25.755839 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753187    2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:52:25.755839 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753189    2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:52:25.755839 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753192    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:52:25.755839 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753195    2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:52:25.755839 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753198    2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:52:25.755839 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753201    2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:52:25.755839 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753204    2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:52:25.755839 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753206    2587 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:52:25.756331 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753209    2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:52:25.756331 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753212    2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:52:25.756331 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753214    2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:52:25.756331 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753217    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:52:25.756331 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753220    2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:52:25.756331 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753222    2587 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:52:25.756331 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753225    2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:52:25.756331 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753227    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:52:25.756331 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753230    2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:52:25.756331 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753233    2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:52:25.756331 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753235    2587 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:52:25.756331 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753238    2587 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:52:25.756331 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753241    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:52:25.756331 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753243    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:52:25.756331 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753246    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:52:25.756331 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753249    2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:52:25.756331 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753252    2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:52:25.756331 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753255    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:52:25.756331 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753257    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:52:25.756331 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753260    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:52:25.756819 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753263    2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:52:25.756819 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753266    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:52:25.756819 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753268    2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:52:25.756819 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753271    2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:52:25.756819 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753273    2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:52:25.756819 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753276    2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:52:25.756819 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753278    2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:52:25.756819 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753281    2587 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:52:25.756819 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753284    2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:52:25.756819 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753286    2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:52:25.756819 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753289    2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:52:25.756819 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753292    2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:52:25.756819 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753295    2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:52:25.756819 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753297    2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:52:25.756819 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753300    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:52:25.756819 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753303    2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:52:25.756819 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753306    2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:52:25.756819 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753308    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:52:25.756819 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753311    2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:52:25.756819 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753313    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:52:25.757294 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753316    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:52:25.757294 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753319    2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:52:25.757294 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753321    2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:52:25.757294 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753324    2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:52:25.757294 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753330    2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:52:25.757294 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753332    2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:52:25.757294 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753335    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:52:25.757294 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753338    2587 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:52:25.757294 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753342    2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:52:25.757294 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753345    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:52:25.757294 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753348    2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:52:25.757294 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753351    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:52:25.757294 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753353    2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:52:25.757294 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753357    2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:52:25.757294 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753359    2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:52:25.757294 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753362    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:52:25.757294 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753365    2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:52:25.757711 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.753370    2587 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 21:52:25.757711 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753513    2587 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 21:52:25.757711 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753519    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 21:52:25.757711 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753522    2587 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 21:52:25.757711 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753525    2587 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 21:52:25.757711 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753528    2587 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 21:52:25.757711 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753531    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 21:52:25.757711 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753534    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 21:52:25.757711 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753536    2587 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 21:52:25.757711 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753539    2587 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 21:52:25.757711 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753541    2587 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 21:52:25.757711 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753545    2587 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 21:52:25.757711 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753547    2587 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 21:52:25.757711 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753550    2587 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 21:52:25.757711 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753553    2587 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 21:52:25.758087 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753556    2587 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 21:52:25.758087 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753559    2587 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 21:52:25.758087 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753561    2587 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 21:52:25.758087 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753565    2587 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 21:52:25.758087 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753569    2587 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 21:52:25.758087 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753572    2587 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 21:52:25.758087 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753575    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 21:52:25.758087 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753578    2587 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 21:52:25.758087 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753582    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 21:52:25.758087 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753584    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 21:52:25.758087 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753587    2587 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 21:52:25.758087 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753590    2587 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 21:52:25.758087 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753593    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 21:52:25.758087 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753595    2587 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 21:52:25.758087 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753598    2587 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 21:52:25.758087 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753600    2587 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 21:52:25.758087 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753603    2587 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 21:52:25.758087 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753605    2587 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 21:52:25.758087 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753608    2587 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 21:52:25.758544 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753610    2587 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 21:52:25.758544 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753613    2587 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 21:52:25.758544 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753615    2587 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 21:52:25.758544 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753618    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 21:52:25.758544 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753620    2587 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 21:52:25.758544 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753623    2587 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 21:52:25.758544 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753625    2587 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 21:52:25.758544 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753628    2587 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 21:52:25.758544 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753631    2587 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 21:52:25.758544 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753634    2587 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 21:52:25.758544 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753636    2587 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 21:52:25.758544 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753639    2587 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 21:52:25.758544 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753641    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 21:52:25.758544 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753644    2587 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 21:52:25.758544 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753647    2587 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 21:52:25.758544 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753649    2587 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 21:52:25.758544 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753652    2587 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 21:52:25.758544 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753654    2587 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 21:52:25.758544 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753657    2587 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 21:52:25.759017 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753661    2587 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 21:52:25.759017 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753663    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 21:52:25.759017 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753666    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 21:52:25.759017 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753669    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 21:52:25.759017 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753671    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 21:52:25.759017 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753674    2587 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 21:52:25.759017 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753676    2587 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 21:52:25.759017 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753679    2587 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 21:52:25.759017 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753682    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 21:52:25.759017 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753684    2587 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 21:52:25.759017 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753687    2587 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 21:52:25.759017 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753689    2587 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 21:52:25.759017 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753691    2587 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 21:52:25.759017 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753694    2587 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 21:52:25.759017 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753697    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 21:52:25.759017 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753699    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 21:52:25.759017 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753701    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 21:52:25.759017 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753704    2587 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 21:52:25.759017 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753707    2587 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 21:52:25.759017 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753710    2587 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 21:52:25.759489 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753713    2587 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 21:52:25.759489 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753715    2587 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 21:52:25.759489 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753718    2587 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 21:52:25.759489 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753720    2587 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 21:52:25.759489 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753723    2587 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 21:52:25.759489 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753725    2587 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 21:52:25.759489 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753728    2587 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 21:52:25.759489 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753730    2587 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 21:52:25.759489 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753733    2587 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 21:52:25.759489 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753735    2587 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 21:52:25.759489 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753738    2587 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 21:52:25.759489 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753741    2587 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 21:52:25.759489 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753743    2587 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 21:52:25.759489 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753746    2587 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 21:52:25.759489 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753749    2587 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 21:52:25.759489 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753753    2587 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 21:52:25.759489 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753756    2587 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 21:52:25.759489 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753759    2587 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 21:52:25.759489 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753762    2587 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 21:52:25.759489 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753765    2587 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 21:52:25.759985 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753767    2587 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 21:52:25.759985 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753770    2587 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 21:52:25.759985 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753773    2587 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 21:52:25.759985 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753776    2587 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 21:52:25.759985 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753778    2587 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 21:52:25.759985 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753781    2587 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 21:52:25.759985 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:25.753784    2587 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 21:52:25.759985 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.753789    2587 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 21:52:25.759985 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.753892    2587 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 21:52:25.759985 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.758312    2587 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 21:52:25.759985 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.759375    2587 server.go:1019] "Starting client certificate rotation"
Sep 11 21:52:25.759985 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.759469    2587 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 21:52:25.759985 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.759513    2587 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 21:52:25.762044 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.762028    2587 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 21:52:25.763206 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.763180    2587 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 21:52:25.780087 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.780069    2587 log.go:25] "Validated CRI v1 runtime API"
Sep 11 21:52:25.783364 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.783347    2587 log.go:25] "Validated CRI v1 image API"
Sep 11 21:52:25.784252 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.784233    2587 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 21:52:25.788325 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.788302    2587 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 bb2bc610-4397-4c0e-a785-532a5320d3b9:/dev/nvme0n1p3 de3e9337-d51b-47a3-a00c-068c26cadee5:/dev/nvme0n1p4]
Sep 11 21:52:25.788401 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.788324    2587 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 21:52:25.791508 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.791353    2587 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 21:52:25.794837 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.794724    2587 manager.go:216] Machine: {Timestamp:2026-09-11 21:52:25.793288036 +0000 UTC m=+0.376194807 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096718 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec259fb4c0eadb965845ab4900dd8a95 SystemUUID:ec259fb4-c0ea-db96-5845-ab4900dd8a95 BootID:99f96c00-1d5c-43d3-8d35-ca914f86b543 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:13:a3:1f:d3:7b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:13:a3:1f:d3:7b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:ed:14:74:10:69 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 21:52:25.794837 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.794826    2587 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 21:52:25.795020 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.794978    2587 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 21:52:25.798368 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.798339    2587 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 21:52:25.798565 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.798370    2587 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-88.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 21:52:25.799964 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.799952    2587 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 21:52:25.800034 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.799970    2587 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 21:52:25.800034 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.799981    2587 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 21:52:25.800034 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.799999    2587 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 21:52:25.801249 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.801237    2587 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 21:52:25.806059 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.806046    2587 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 21:52:25.806190 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.806179    2587 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 21:52:25.806245 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.806217    2587 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 21:52:25.806245 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.806231    2587 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 21:52:25.806341 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.806252    2587 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 21:52:25.806341 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.806266    2587 kubelet.go:398] "Adding apiserver pod source"
Sep 11 21:52:25.806341 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.806280    2587 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 21:52:25.806730 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.806715    2587 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 21:52:25.806808 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.806739    2587 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 21:52:25.809240 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.809219    2587 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 21:52:25.811605 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.811589    2587 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 21:52:25.811684 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.811619    2587 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 21:52:25.811684 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.811642    2587 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c4d4f"
Sep 11 21:52:25.813550 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.813529    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 21:52:25.813550 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.813553    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 21:52:25.813675 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.813603    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 21:52:25.813675 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.813615    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 21:52:25.813675 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.813625    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 21:52:25.813675 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.813634    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 21:52:25.813675 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.813652    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 21:52:25.813675 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.813659    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 21:52:25.813675 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.813667    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 21:52:25.813675 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.813674    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 21:52:25.814009 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.813696    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 21:52:25.814009 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.813703    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 21:52:25.814598 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.814587    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 21:52:25.814598 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.814597    2587 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 21:52:25.815790 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:25.815767    2587 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 21:52:25.815857 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:25.815829    2587 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-88.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 21:52:25.817201 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.817184    2587 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c4d4f"
Sep 11 21:52:25.818889 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.818865    2587 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-88.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 21:52:25.819241 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.819222    2587 server.go:1268] "Started kubelet"
Sep 11 21:52:25.819646 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.819610    2587 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 21:52:25.820306 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.820222    2587 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 21:52:25.820399 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.820335    2587 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 21:52:25.820423 ip-10-0-139-88 systemd[1]: Started Kubernetes Kubelet.
Sep 11 21:52:25.821914 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.821897    2587 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 21:52:25.826901 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:25.826861    2587 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 21:52:25.827850 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.827832    2587 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 21:52:25.831199 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.831173    2587 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 21:52:25.831310 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.831296    2587 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 21:52:25.831401 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:25.831329    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 11 21:52:25.832029 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.832000    2587 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 21:52:25.832123 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.832005    2587 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 21:52:25.832123 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.832113    2587 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 21:52:25.832222 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.832178    2587 factory.go:55] Registering systemd factory
Sep 11 21:52:25.832222 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.832192    2587 factory.go:223] Registration of the systemd container factory successfully
Sep 11 21:52:25.832222 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.832198    2587 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 21:52:25.832222 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.832208    2587 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 21:52:25.832445 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.832430    2587 factory.go:153] Registering CRI-O factory
Sep 11 21:52:25.832545 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.832448    2587 factory.go:223] Registration of the crio container factory successfully
Sep 11 21:52:25.832545 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.832518    2587 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 21:52:25.832545 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.832544    2587 factory.go:103] Registering Raw factory
Sep 11 21:52:25.832683 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.832561    2587 manager.go:1196] Started watching for new ooms in manager
Sep 11 21:52:25.833000 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.832986    2587 manager.go:349] Starting recovery of all containers
Sep 11 21:52:25.833606 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.833586    2587 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:52:25.837766 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:25.837561    2587 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-88.ec2.internal\" not found" node="ip-10-0-139-88.ec2.internal"
Sep 11 21:52:25.843647 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.843632    2587 manager.go:354] Recovery completed
Sep 11 21:52:25.848114 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.848097    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:52:25.849840 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.849821    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:52:25.849921 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.849854    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:52:25.849921 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.849869    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:52:25.850422 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.850409    2587 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 21:52:25.850422 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.850419    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:52:25.850422 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.850421    2587 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 21:52:25.850595 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.850437    2587 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 21:52:25.851030 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.851018    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:52:25.851088 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.851043    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:52:25.851088 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.851055    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:52:25.852673 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.852660    2587 policy_none.go:49] "None policy: Start"
Sep 11 21:52:25.852726 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.852676    2587 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 21:52:25.852726 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.852686    2587 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 21:52:25.854644 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.854632    2587 policy_none.go:47] "Start"
Sep 11 21:52:25.920170 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.920138    2587 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 21:52:25.922480 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.921237    2587 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 21:52:25.922480 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.921286    2587 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 21:52:25.922480 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.921317    2587 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 21:52:25.922480 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:25.921366    2587 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 11 21:52:25.923735 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.923717    2587 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:52:25.931691 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:25.931671    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 11 21:52:25.997182 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.997128    2587 manager.go:337] "Starting Device Plugin manager"
Sep 11 21:52:25.997182 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:25.997178    2587 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 21:52:25.997343 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.997192    2587 server.go:87] "Starting device plugin registration server"
Sep 11 21:52:25.997466 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.997455    2587 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 21:52:25.997530 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.997468    2587 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 21:52:25.997584 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.997566    2587 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 21:52:25.997658 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.997642    2587 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 21:52:25.997658 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:25.997655    2587 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 21:52:25.998447 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:25.998430    2587 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 21:52:25.998536 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:25.998473    2587 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 11 21:52:26.022313 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.021659    2587 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-88.ec2.internal"]
Sep 11 21:52:26.022424 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.022361    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:52:26.023652 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.023635    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:52:26.023755 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.023670    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:52:26.023755 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.023680    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:52:26.026073 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.026061    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:52:26.026191 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.026174    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal"
Sep 11 21:52:26.026247 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.026216    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:52:26.026783 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.026770    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:52:26.026783 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.026778    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:52:26.026906 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.026792    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:52:26.026906 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.026796    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:52:26.026906 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.026807    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:52:26.026988 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.026806    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:52:26.029149 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.029133    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-88.ec2.internal"
Sep 11 21:52:26.029234 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.029158    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:52:26.029825 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.029812    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:52:26.029884 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.029839    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:52:26.029884 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.029849    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:52:26.032967 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.032952    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1f90deb2b4826b854b012a456c786bb2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal\" (UID: \"1f90deb2b4826b854b012a456c786bb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal"
Sep 11 21:52:26.033029 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.032975    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f90deb2b4826b854b012a456c786bb2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal\" (UID: \"1f90deb2b4826b854b012a456c786bb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal"
Sep 11 21:52:26.033029 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.032993    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d56814d1b26653aa054a4eb19dfe6290-config\") pod \"kube-apiserver-proxy-ip-10-0-139-88.ec2.internal\" (UID: \"d56814d1b26653aa054a4eb19dfe6290\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-88.ec2.internal"
Sep 11 21:52:26.064260 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:26.064228    2587 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-88.ec2.internal\" not found" node="ip-10-0-139-88.ec2.internal"
Sep 11 21:52:26.068580 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:26.068565    2587 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-88.ec2.internal\" not found" node="ip-10-0-139-88.ec2.internal"
Sep 11 21:52:26.098229 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.098209    2587 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 21:52:26.099105 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.099092    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 21:52:26.099172 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.099120    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 21:52:26.099172 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.099144    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeHasSufficientPID"
Sep 11 21:52:26.099172 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.099169    2587 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-88.ec2.internal"
Sep 11 21:52:26.105270 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.105250    2587 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-88.ec2.internal"
Sep 11 21:52:26.105366 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:26.105276    2587 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-88.ec2.internal\": node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 11 21:52:26.119285 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:26.119256    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 11 21:52:26.133128 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.133097    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1f90deb2b4826b854b012a456c786bb2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal\" (UID: \"1f90deb2b4826b854b012a456c786bb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal"
Sep 11 21:52:26.133248 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.133136    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f90deb2b4826b854b012a456c786bb2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal\" (UID: \"1f90deb2b4826b854b012a456c786bb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal"
Sep 11 21:52:26.133248 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.133163    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d56814d1b26653aa054a4eb19dfe6290-config\") pod \"kube-apiserver-proxy-ip-10-0-139-88.ec2.internal\" (UID: \"d56814d1b26653aa054a4eb19dfe6290\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-88.ec2.internal"
Sep 11 21:52:26.133248 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.133216    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d56814d1b26653aa054a4eb19dfe6290-config\") pod \"kube-apiserver-proxy-ip-10-0-139-88.ec2.internal\" (UID: \"d56814d1b26653aa054a4eb19dfe6290\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-88.ec2.internal"
Sep 11 21:52:26.133248 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.133241    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1f90deb2b4826b854b012a456c786bb2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal\" (UID: \"1f90deb2b4826b854b012a456c786bb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal"
Sep 11 21:52:26.133390 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.133207    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f90deb2b4826b854b012a456c786bb2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal\" (UID: \"1f90deb2b4826b854b012a456c786bb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal"
Sep 11 21:52:26.220297 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:26.220265    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 11 21:52:26.321023 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:26.320963    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 11 21:52:26.366450 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.366407    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal"
Sep 11 21:52:26.372483 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:26.372465    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-88.ec2.internal"
Sep 11 21:52:27.829934 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:27.829898    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 11 21:52:27.930357 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:27.930314    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 11 21:52:28.030954 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:28.030869    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 11 21:52:28.131411 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:28.131374    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 11 21:52:28.166909 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:28.166881    2587 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 21:52:28.167579 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:28.167039    2587 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 21:52:28.167579 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:28.167059    2587 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 21:52:28.231778 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:28.231608    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 11 21:52:28.235515 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:28.235467    2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 21:47:27 +0000 UTC" deadline="2028-02-23 21:37:24.186247576 +0000 UTC"
Sep 11 21:52:28.235515 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:28.235517    2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12719h44m55.950736332s"
Sep 11 21:52:28.331828 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:28.331800    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 11 21:52:28.333677 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:28.333649    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f90deb2b4826b854b012a456c786bb2.slice/crio-faf7c1b5396c24fbb1b2440d79f1de0f0590f90c835b1131db7805bfd1881bdf WatchSource:0}: Error finding container faf7c1b5396c24fbb1b2440d79f1de0f0590f90c835b1131db7805bfd1881bdf: Status 404 returned error can't find the container with id faf7c1b5396c24fbb1b2440d79f1de0f0590f90c835b1131db7805bfd1881bdf
Sep 11 21:52:28.335246 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:28.335224    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56814d1b26653aa054a4eb19dfe6290.slice/crio-3fef67ea13215893684d04021b3e9ec89b6c32affaf9dcd622f556a9ad2af2ab WatchSource:0}: Error finding container 3fef67ea13215893684d04021b3e9ec89b6c32affaf9dcd622f556a9ad2af2ab: Status 404 returned error can't find the container with id 3fef67ea13215893684d04021b3e9ec89b6c32affaf9dcd622f556a9ad2af2ab
Sep 11 21:52:28.339850 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:28.339833    2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 21:52:28.432814 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:28.432773    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 11 21:52:28.533339 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:28.533312    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 11 21:52:28.633968 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:28.633872    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 11 21:52:28.709240 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:28.709202    2587 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:52:28.734722 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:28.734687    2587 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-88.ec2.internal\" not found"
Sep 11 21:52:28.792079 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:28.792048    2587 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:52:28.840187 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:28.840155    2587 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal"
Sep 11 21:52:28.851659 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:28.851552    2587 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 21:52:28.852822 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:28.852606    2587 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-88.ec2.internal"
Sep 11 21:52:28.859163 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:28.859140    2587 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 21:52:28.990517 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:28.990416    2587 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:52:29.215696 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.215667    2587 apiserver.go:52] "Watching apiserver"
Sep 11 21:52:29.225382 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.225355    2587 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 21:52:29.225749 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.225727    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-9j5p9","kube-system/kube-apiserver-proxy-ip-10-0-139-88.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs","openshift-dns/node-resolver-756n4","openshift-multus/multus-492xh","openshift-multus/multus-additional-cni-plugins-lzrhd","openshift-multus/network-metrics-daemon-n87wl","openshift-network-operator/iptables-alerter-q4cwb","kube-system/global-pull-secret-syncer-fc5j4","openshift-cluster-node-tuning-operator/tuned-zs6pz","openshift-image-registry/node-ca-4vqqq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal","openshift-network-diagnostics/network-check-target-hxztv","openshift-ovn-kubernetes/ovnkube-node-lw6qf"]
Sep 11 21:52:29.228245 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.228227    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:29.228373 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:29.228304    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n87wl" podUID="92a0df5e-f5d1-4331-9ee2-7ed9916f00fd"
Sep 11 21:52:29.230414 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.230394    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:52:29.230533 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:29.230469    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxztv" podUID="3752cc39-b860-4328-82d5-d67850cd906c"
Sep 11 21:52:29.232609 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.232588    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs"
Sep 11 21:52:29.235636 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.235612    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-756n4"
Sep 11 21:52:29.237497 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.237439    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.238284 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.238026    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 21:52:29.238284 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.238106    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 21:52:29.238284 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.238121    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-r5nrg\""
Sep 11 21:52:29.238284 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.238164    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 21:52:29.240047 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.240030    2587 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 21:52:29.240282 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.240249    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lzrhd"
Sep 11 21:52:29.240875 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.240828    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-24tmj\""
Sep 11 21:52:29.240962 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.240891    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 21:52:29.240962 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.240946    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 21:52:29.241212 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.241131    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 21:52:29.242645 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.242290    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 21:52:29.242645 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.242311    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 21:52:29.242645 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.242411    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4mr4m\""
Sep 11 21:52:29.242645 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.242545    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 21:52:29.242892 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.242656    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:29.242892 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:29.242713    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc5j4" podUID="05523d8a-2c9a-47f3-ad9d-4c508546cd42"
Sep 11 21:52:29.244130 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.243294    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 21:52:29.244130 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.243419    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 21:52:29.244130 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.243632    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xbbr2\""
Sep 11 21:52:29.244826 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.244779    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9j5p9"
Sep 11 21:52:29.247159 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.247141    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.247484 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.247465    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 21:52:29.247854 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.247835    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 21:52:29.247943 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.247919    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2gpf6\""
Sep 11 21:52:29.249564 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.249543    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-metrics-certs\") pod \"network-metrics-daemon-n87wl\" (UID: \"92a0df5e-f5d1-4331-9ee2-7ed9916f00fd\") " pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:29.249654 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.249577    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.249654 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.249599    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-556tv\" (UniqueName: \"kubernetes.io/projected/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-kube-api-access-556tv\") pod \"network-metrics-daemon-n87wl\" (UID: \"92a0df5e-f5d1-4331-9ee2-7ed9916f00fd\") " pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:29.249654 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.249628    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5d5e4814-3128-4082-aca2-5bbfa715b2e4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lzrhd\" (UID: \"5d5e4814-3128-4082-aca2-5bbfa715b2e4\") " pod="openshift-multus/multus-additional-cni-plugins-lzrhd"
Sep 11 21:52:29.249811 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.249686    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/50fccc19-46f5-4bdc-8e4d-c728752de971-hosts-file\") pod \"node-resolver-756n4\" (UID: \"50fccc19-46f5-4bdc-8e4d-c728752de971\") " pod="openshift-dns/node-resolver-756n4"
Sep 11 21:52:29.249811 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.249712    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5d8e96d2-170d-48ab-bf18-e43b69834d56-multus-daemon-config\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.249811 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.249736    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-etc-kubernetes\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.249811 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.249762    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e4814-3128-4082-aca2-5bbfa715b2e4-system-cni-dir\") pod \"multus-additional-cni-plugins-lzrhd\" (UID: \"5d5e4814-3128-4082-aca2-5bbfa715b2e4\") " pod="openshift-multus/multus-additional-cni-plugins-lzrhd"
Sep 11 21:52:29.249811 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.249785    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e4814-3128-4082-aca2-5bbfa715b2e4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lzrhd\" (UID: \"5d5e4814-3128-4082-aca2-5bbfa715b2e4\") " pod="openshift-multus/multus-additional-cni-plugins-lzrhd"
Sep 11 21:52:29.249811 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.249809    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d8e96d2-170d-48ab-bf18-e43b69834d56-cni-binary-copy\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.250054 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.249833    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cb2a378c-7071-4cb8-9588-5993651d056f-etc-selinux\") pod \"aws-ebs-csi-driver-node-bndqs\" (UID: \"cb2a378c-7071-4cb8-9588-5993651d056f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs"
Sep 11 21:52:29.250054 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.249855    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-system-cni-dir\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.250054 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.249882    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-multus-cni-dir\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.250054 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.249905    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-multus-socket-dir-parent\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.250054 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.249958    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9tqb\" (UniqueName: \"kubernetes.io/projected/5d5e4814-3128-4082-aca2-5bbfa715b2e4-kube-api-access-j9tqb\") pod \"multus-additional-cni-plugins-lzrhd\" (UID: \"5d5e4814-3128-4082-aca2-5bbfa715b2e4\") " pod="openshift-multus/multus-additional-cni-plugins-lzrhd"
Sep 11 21:52:29.250054 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.249989    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/05523d8a-2c9a-47f3-ad9d-4c508546cd42-kubelet-config\") pod \"global-pull-secret-syncer-fc5j4\" (UID: \"05523d8a-2c9a-47f3-ad9d-4c508546cd42\") " pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:29.250054 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250007    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cb2a378c-7071-4cb8-9588-5993651d056f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bndqs\" (UID: \"cb2a378c-7071-4cb8-9588-5993651d056f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs"
Sep 11 21:52:29.250054 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250029    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-host-run-netns\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.250319 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250057    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 21:52:29.250319 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250067    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsnn2\" (UniqueName: \"kubernetes.io/projected/5d8e96d2-170d-48ab-bf18-e43b69834d56-kube-api-access-tsnn2\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.250319 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250090    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/05523d8a-2c9a-47f3-ad9d-4c508546cd42-dbus\") pod \"global-pull-secret-syncer-fc5j4\" (UID: \"05523d8a-2c9a-47f3-ad9d-4c508546cd42\") " pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:29.250319 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250113    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05523d8a-2c9a-47f3-ad9d-4c508546cd42-original-pull-secret\") pod \"global-pull-secret-syncer-fc5j4\" (UID: \"05523d8a-2c9a-47f3-ad9d-4c508546cd42\") " pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:29.250319 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250131    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fccc19-46f5-4bdc-8e4d-c728752de971-tmp-dir\") pod \"node-resolver-756n4\" (UID: \"50fccc19-46f5-4bdc-8e4d-c728752de971\") " pod="openshift-dns/node-resolver-756n4"
Sep 11 21:52:29.250319 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250149    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-cnibin\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.250319 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250167    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-host-run-k8s-cni-cncf-io\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.250319 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250183    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-host-var-lib-kubelet\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.250319 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250196    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-hostroot\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.250319 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250234    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-multus-conf-dir\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.250319 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250257    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d5e4814-3128-4082-aca2-5bbfa715b2e4-cnibin\") pod \"multus-additional-cni-plugins-lzrhd\" (UID: \"5d5e4814-3128-4082-aca2-5bbfa715b2e4\") " pod="openshift-multus/multus-additional-cni-plugins-lzrhd"
Sep 11 21:52:29.250319 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250280    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d5e4814-3128-4082-aca2-5bbfa715b2e4-cni-binary-copy\") pod \"multus-additional-cni-plugins-lzrhd\" (UID: \"5d5e4814-3128-4082-aca2-5bbfa715b2e4\") " pod="openshift-multus/multus-additional-cni-plugins-lzrhd"
Sep 11 21:52:29.250319 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250295    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5d5e4814-3128-4082-aca2-5bbfa715b2e4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lzrhd\" (UID: \"5d5e4814-3128-4082-aca2-5bbfa715b2e4\") " pod="openshift-multus/multus-additional-cni-plugins-lzrhd"
Sep 11 21:52:29.250319 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250309    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4vnl\" (UniqueName: \"kubernetes.io/projected/3752cc39-b860-4328-82d5-d67850cd906c-kube-api-access-b4vnl\") pod \"network-check-target-hxztv\" (UID: \"3752cc39-b860-4328-82d5-d67850cd906c\") " pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:52:29.250319 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250346    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cb2a378c-7071-4cb8-9588-5993651d056f-socket-dir\") pod \"aws-ebs-csi-driver-node-bndqs\" (UID: \"cb2a378c-7071-4cb8-9588-5993651d056f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs"
Sep 11 21:52:29.250906 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250361    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cb2a378c-7071-4cb8-9588-5993651d056f-registration-dir\") pod \"aws-ebs-csi-driver-node-bndqs\" (UID: \"cb2a378c-7071-4cb8-9588-5993651d056f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs"
Sep 11 21:52:29.250906 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250387    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cb2a378c-7071-4cb8-9588-5993651d056f-device-dir\") pod \"aws-ebs-csi-driver-node-bndqs\" (UID: \"cb2a378c-7071-4cb8-9588-5993651d056f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs"
Sep 11 21:52:29.250906 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250407    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cb2a378c-7071-4cb8-9588-5993651d056f-sys-fs\") pod \"aws-ebs-csi-driver-node-bndqs\" (UID: \"cb2a378c-7071-4cb8-9588-5993651d056f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs"
Sep 11 21:52:29.250906 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250428    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-host-var-lib-cni-bin\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.250906 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250451    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-host-var-lib-cni-multus\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.250906 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250485    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-host-run-multus-certs\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.250906 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250508    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9fcx\" (UniqueName: \"kubernetes.io/projected/cb2a378c-7071-4cb8-9588-5993651d056f-kube-api-access-t9fcx\") pod \"aws-ebs-csi-driver-node-bndqs\" (UID: \"cb2a378c-7071-4cb8-9588-5993651d056f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs"
Sep 11 21:52:29.250906 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250547    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqf2b\" (UniqueName: \"kubernetes.io/projected/50fccc19-46f5-4bdc-8e4d-c728752de971-kube-api-access-gqf2b\") pod \"node-resolver-756n4\" (UID: \"50fccc19-46f5-4bdc-8e4d-c728752de971\") " pod="openshift-dns/node-resolver-756n4"
Sep 11 21:52:29.251314 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.250939    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 21:52:29.251314 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.251196    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 21:52:29.251412 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.251370    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 21:52:29.252194 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.252174    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 21:52:29.252194 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.252186    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 21:52:29.252458 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.252188    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tscts\""
Sep 11 21:52:29.252532 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.252518    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 21:52:29.252638 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.252621    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kx7x5\""
Sep 11 21:52:29.252782 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.252769    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4vqqq"
Sep 11 21:52:29.253085 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.252861    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:52:29.255118 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.255081    2587 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 21:52:29.255211 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.255138    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q4cwb"
Sep 11 21:52:29.255294 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.255232    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 21:52:29.255425 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.255404    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 21:52:29.255852 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.255691    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 21:52:29.255852 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.255769    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5x992\""
Sep 11 21:52:29.257788 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.257771    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6lrps\""
Sep 11 21:52:29.257865 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.257792    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 21:52:29.258253 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.258063    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 21:52:29.258253 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.258162    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 21:52:29.279091 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.279054    2587 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xfnqr"
Sep 11 21:52:29.284892 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.284865    2587 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xfnqr"
Sep 11 21:52:29.334762 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.334702    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-88.ec2.internal" event={"ID":"d56814d1b26653aa054a4eb19dfe6290","Type":"ContainerStarted","Data":"3fef67ea13215893684d04021b3e9ec89b6c32affaf9dcd622f556a9ad2af2ab"}
Sep 11 21:52:29.335966 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.335936    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal" event={"ID":"1f90deb2b4826b854b012a456c786bb2","Type":"ContainerStarted","Data":"faf7c1b5396c24fbb1b2440d79f1de0f0590f90c835b1131db7805bfd1881bdf"}
Sep 11 21:52:29.341511 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.341481    2587 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 21:52:29.351295 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.351273    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-ovnkube-config\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.351430 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.351309    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-host-run-multus-certs\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.351430 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.351347    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9fcx\" (UniqueName: \"kubernetes.io/projected/cb2a378c-7071-4cb8-9588-5993651d056f-kube-api-access-t9fcx\") pod \"aws-ebs-csi-driver-node-bndqs\" (UID: \"cb2a378c-7071-4cb8-9588-5993651d056f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs"
Sep 11 21:52:29.351430 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.351370    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqf2b\" (UniqueName: \"kubernetes.io/projected/50fccc19-46f5-4bdc-8e4d-c728752de971-kube-api-access-gqf2b\") pod \"node-resolver-756n4\" (UID: \"50fccc19-46f5-4bdc-8e4d-c728752de971\") " pod="openshift-dns/node-resolver-756n4"
Sep 11 21:52:29.351430 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.351394    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlwcq\" (UniqueName: \"kubernetes.io/projected/8e0307d0-2de3-46fe-a794-43536ce8eb60-kube-api-access-wlwcq\") pod \"iptables-alerter-q4cwb\" (UID: \"8e0307d0-2de3-46fe-a794-43536ce8eb60\") " pod="openshift-network-operator/iptables-alerter-q4cwb"
Sep 11 21:52:29.351430 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.351395    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-host-run-multus-certs\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.351430 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.351418    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-host-slash\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.351688 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.351532    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-ovn-node-metrics-cert\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.351688 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.351569    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm4ck\" (UniqueName: \"kubernetes.io/projected/741f47df-f402-494f-a78e-03fb0bc10cac-kube-api-access-rm4ck\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.351688 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.351597    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-node-log\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.351688 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.351638    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5d8e96d2-170d-48ab-bf18-e43b69834d56-multus-daemon-config\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.351688 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.351662    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-etc-kubernetes\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.351688 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.351685    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e4814-3128-4082-aca2-5bbfa715b2e4-system-cni-dir\") pod \"multus-additional-cni-plugins-lzrhd\" (UID: \"5d5e4814-3128-4082-aca2-5bbfa715b2e4\") " pod="openshift-multus/multus-additional-cni-plugins-lzrhd"
Sep 11 21:52:29.351910 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.351708    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d8e96d2-170d-48ab-bf18-e43b69834d56-cni-binary-copy\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.351910 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.351733    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-etc-sysconfig\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.351910 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.351739    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-etc-kubernetes\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.351910 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.351763    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e4814-3128-4082-aca2-5bbfa715b2e4-system-cni-dir\") pod \"multus-additional-cni-plugins-lzrhd\" (UID: \"5d5e4814-3128-4082-aca2-5bbfa715b2e4\") " pod="openshift-multus/multus-additional-cni-plugins-lzrhd"
Sep 11 21:52:29.351910 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.351865    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-run\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.351910 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.351904    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-system-cni-dir\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.352211 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.351930    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-multus-cni-dir\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.352211 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.351957    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-multus-socket-dir-parent\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.352211 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.351981    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cb2a378c-7071-4cb8-9588-5993651d056f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bndqs\" (UID: \"cb2a378c-7071-4cb8-9588-5993651d056f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs"
Sep 11 21:52:29.352211 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.351992    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-system-cni-dir\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.352211 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352005    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/87753932-baf9-409d-948e-3d9eb12b1095-serviceca\") pod \"node-ca-4vqqq\" (UID: \"87753932-baf9-409d-948e-3d9eb12b1095\") " pod="openshift-image-registry/node-ca-4vqqq"
Sep 11 21:52:29.352211 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352036    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb56d2f4-9566-4726-81dd-39cf43dbbc42-agent-certs\") pod \"konnectivity-agent-9j5p9\" (UID: \"fb56d2f4-9566-4726-81dd-39cf43dbbc42\") " pod="kube-system/konnectivity-agent-9j5p9"
Sep 11 21:52:29.352211 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352061    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-systemd-units\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.352211 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352070    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cb2a378c-7071-4cb8-9588-5993651d056f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bndqs\" (UID: \"cb2a378c-7071-4cb8-9588-5993651d056f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs"
Sep 11 21:52:29.352211 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352086    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-host-cni-bin\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.352211 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352094    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-multus-cni-dir\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.352211 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352109    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-etc-systemd\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.352211 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352122    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-multus-socket-dir-parent\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.352211 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352154    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/741f47df-f402-494f-a78e-03fb0bc10cac-etc-tuned\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.352211 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352196    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/741f47df-f402-494f-a78e-03fb0bc10cac-tmp\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.352699 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352224    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsnn2\" (UniqueName: \"kubernetes.io/projected/5d8e96d2-170d-48ab-bf18-e43b69834d56-kube-api-access-tsnn2\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.352699 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352250    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/05523d8a-2c9a-47f3-ad9d-4c508546cd42-dbus\") pod \"global-pull-secret-syncer-fc5j4\" (UID: \"05523d8a-2c9a-47f3-ad9d-4c508546cd42\") " pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:29.352699 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352272    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fccc19-46f5-4bdc-8e4d-c728752de971-tmp-dir\") pod \"node-resolver-756n4\" (UID: \"50fccc19-46f5-4bdc-8e4d-c728752de971\") " pod="openshift-dns/node-resolver-756n4"
Sep 11 21:52:29.352699 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352296    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-log-socket\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.352699 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352305    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d8e96d2-170d-48ab-bf18-e43b69834d56-cni-binary-copy\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.352699 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352320    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-var-lib-kubelet\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.352699 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352360    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-cnibin\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.352699 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352383    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-host-var-lib-kubelet\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.352699 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352393    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/05523d8a-2c9a-47f3-ad9d-4c508546cd42-dbus\") pod \"global-pull-secret-syncer-fc5j4\" (UID: \"05523d8a-2c9a-47f3-ad9d-4c508546cd42\") " pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:29.352699 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352406    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-hostroot\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.352699 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352428    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-multus-conf-dir\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.352699 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352305    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5d8e96d2-170d-48ab-bf18-e43b69834d56-multus-daemon-config\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.352699 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352466    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-host-var-lib-kubelet\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.352699 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352496    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d5e4814-3128-4082-aca2-5bbfa715b2e4-cni-binary-copy\") pod \"multus-additional-cni-plugins-lzrhd\" (UID: \"5d5e4814-3128-4082-aca2-5bbfa715b2e4\") " pod="openshift-multus/multus-additional-cni-plugins-lzrhd"
Sep 11 21:52:29.352699 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352495    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-hostroot\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.352699 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352520    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-multus-conf-dir\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.352699 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352527    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4vnl\" (UniqueName: \"kubernetes.io/projected/3752cc39-b860-4328-82d5-d67850cd906c-kube-api-access-b4vnl\") pod \"network-check-target-hxztv\" (UID: \"3752cc39-b860-4328-82d5-d67850cd906c\") " pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:52:29.352699 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352561    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cb2a378c-7071-4cb8-9588-5993651d056f-socket-dir\") pod \"aws-ebs-csi-driver-node-bndqs\" (UID: \"cb2a378c-7071-4cb8-9588-5993651d056f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs"
Sep 11 21:52:29.353463 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352583    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50fccc19-46f5-4bdc-8e4d-c728752de971-tmp-dir\") pod \"node-resolver-756n4\" (UID: \"50fccc19-46f5-4bdc-8e4d-c728752de971\") " pod="openshift-dns/node-resolver-756n4"
Sep 11 21:52:29.353463 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352589    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-etc-modprobe-d\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.353463 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352564    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-cnibin\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.353463 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352612    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-lib-modules\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.353463 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352642    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghbk9\" (UniqueName: \"kubernetes.io/projected/87753932-baf9-409d-948e-3d9eb12b1095-kube-api-access-ghbk9\") pod \"node-ca-4vqqq\" (UID: \"87753932-baf9-409d-948e-3d9eb12b1095\") " pod="openshift-image-registry/node-ca-4vqqq"
Sep 11 21:52:29.353463 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352667    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb56d2f4-9566-4726-81dd-39cf43dbbc42-konnectivity-ca\") pod \"konnectivity-agent-9j5p9\" (UID: \"fb56d2f4-9566-4726-81dd-39cf43dbbc42\") " pod="kube-system/konnectivity-agent-9j5p9"
Sep 11 21:52:29.353463 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352698    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-host-kubelet\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.353463 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352713    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cb2a378c-7071-4cb8-9588-5993651d056f-socket-dir\") pod \"aws-ebs-csi-driver-node-bndqs\" (UID: \"cb2a378c-7071-4cb8-9588-5993651d056f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs"
Sep 11 21:52:29.353463 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352722    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-var-lib-openvswitch\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.353463 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352757    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-ovnkube-script-lib\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.353463 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352788    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-host-var-lib-cni-bin\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.353463 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352814    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-host-var-lib-cni-multus\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.353463 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352823    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-host-var-lib-cni-bin\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.353463 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352837    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-etc-openvswitch\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.353463 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352856    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-host-var-lib-cni-multus\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.353463 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352861    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-host-cni-netd\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.353463 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352887    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-metrics-certs\") pod \"network-metrics-daemon-n87wl\" (UID: \"92a0df5e-f5d1-4331-9ee2-7ed9916f00fd\") " pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:29.354164 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352910    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-556tv\" (UniqueName: \"kubernetes.io/projected/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-kube-api-access-556tv\") pod \"network-metrics-daemon-n87wl\" (UID: \"92a0df5e-f5d1-4331-9ee2-7ed9916f00fd\") " pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:29.354164 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352938    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5d5e4814-3128-4082-aca2-5bbfa715b2e4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lzrhd\" (UID: \"5d5e4814-3128-4082-aca2-5bbfa715b2e4\") " pod="openshift-multus/multus-additional-cni-plugins-lzrhd"
Sep 11 21:52:29.354164 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352964    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/50fccc19-46f5-4bdc-8e4d-c728752de971-hosts-file\") pod \"node-resolver-756n4\" (UID: \"50fccc19-46f5-4bdc-8e4d-c728752de971\") " pod="openshift-dns/node-resolver-756n4"
Sep 11 21:52:29.354164 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.352988    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-etc-sysctl-conf\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.354164 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:29.353000    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:29.354164 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353012    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-run-openvswitch\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.354164 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353039    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-host-run-ovn-kubernetes\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.354164 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:29.353059    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-metrics-certs podName:92a0df5e-f5d1-4331-9ee2-7ed9916f00fd nodeName:}" failed. No retries permitted until 2026-09-11 21:52:29.853039689 +0000 UTC m=+3.028116059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-metrics-certs") pod "network-metrics-daemon-n87wl" (UID: "92a0df5e-f5d1-4331-9ee2-7ed9916f00fd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:29.354164 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353073    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d5e4814-3128-4082-aca2-5bbfa715b2e4-cni-binary-copy\") pod \"multus-additional-cni-plugins-lzrhd\" (UID: \"5d5e4814-3128-4082-aca2-5bbfa715b2e4\") " pod="openshift-multus/multus-additional-cni-plugins-lzrhd"
Sep 11 21:52:29.354164 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353092    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/50fccc19-46f5-4bdc-8e4d-c728752de971-hosts-file\") pod \"node-resolver-756n4\" (UID: \"50fccc19-46f5-4bdc-8e4d-c728752de971\") " pod="openshift-dns/node-resolver-756n4"
Sep 11 21:52:29.354164 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353127    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-env-overrides\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.354164 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353153    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e4814-3128-4082-aca2-5bbfa715b2e4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lzrhd\" (UID: \"5d5e4814-3128-4082-aca2-5bbfa715b2e4\") " pod="openshift-multus/multus-additional-cni-plugins-lzrhd"
Sep 11 21:52:29.354164 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353168    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-sys\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.354164 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353188    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-host\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.354164 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353210    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8e0307d0-2de3-46fe-a794-43536ce8eb60-iptables-alerter-script\") pod \"iptables-alerter-q4cwb\" (UID: \"8e0307d0-2de3-46fe-a794-43536ce8eb60\") " pod="openshift-network-operator/iptables-alerter-q4cwb"
Sep 11 21:52:29.354164 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353233    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vf5p\" (UniqueName: \"kubernetes.io/projected/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-kube-api-access-2vf5p\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.354729 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353257    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cb2a378c-7071-4cb8-9588-5993651d056f-etc-selinux\") pod \"aws-ebs-csi-driver-node-bndqs\" (UID: \"cb2a378c-7071-4cb8-9588-5993651d056f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs"
Sep 11 21:52:29.354729 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353278    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-etc-kubernetes\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.354729 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353311    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-run-ovn\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.354729 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353315    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e4814-3128-4082-aca2-5bbfa715b2e4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lzrhd\" (UID: \"5d5e4814-3128-4082-aca2-5bbfa715b2e4\") " pod="openshift-multus/multus-additional-cni-plugins-lzrhd"
Sep 11 21:52:29.354729 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353362    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cb2a378c-7071-4cb8-9588-5993651d056f-etc-selinux\") pod \"aws-ebs-csi-driver-node-bndqs\" (UID: \"cb2a378c-7071-4cb8-9588-5993651d056f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs"
Sep 11 21:52:29.354729 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353365    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9tqb\" (UniqueName: \"kubernetes.io/projected/5d5e4814-3128-4082-aca2-5bbfa715b2e4-kube-api-access-j9tqb\") pod \"multus-additional-cni-plugins-lzrhd\" (UID: \"5d5e4814-3128-4082-aca2-5bbfa715b2e4\") " pod="openshift-multus/multus-additional-cni-plugins-lzrhd"
Sep 11 21:52:29.354729 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353412    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/05523d8a-2c9a-47f3-ad9d-4c508546cd42-kubelet-config\") pod \"global-pull-secret-syncer-fc5j4\" (UID: \"05523d8a-2c9a-47f3-ad9d-4c508546cd42\") " pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:29.354729 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353440    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-host-run-netns\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.354729 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353466    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.354729 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353452    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5d5e4814-3128-4082-aca2-5bbfa715b2e4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lzrhd\" (UID: \"5d5e4814-3128-4082-aca2-5bbfa715b2e4\") " pod="openshift-multus/multus-additional-cni-plugins-lzrhd"
Sep 11 21:52:29.354729 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353495    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-host-run-netns\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.354729 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353522    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05523d8a-2c9a-47f3-ad9d-4c508546cd42-original-pull-secret\") pod \"global-pull-secret-syncer-fc5j4\" (UID: \"05523d8a-2c9a-47f3-ad9d-4c508546cd42\") " pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:29.354729 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353534    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-host-run-netns\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.354729 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353545    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-etc-sysctl-d\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.354729 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353495    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/05523d8a-2c9a-47f3-ad9d-4c508546cd42-kubelet-config\") pod \"global-pull-secret-syncer-fc5j4\" (UID: \"05523d8a-2c9a-47f3-ad9d-4c508546cd42\") " pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:29.354729 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353571    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-host-run-k8s-cni-cncf-io\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.354729 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353598    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d5e4814-3128-4082-aca2-5bbfa715b2e4-cnibin\") pod \"multus-additional-cni-plugins-lzrhd\" (UID: \"5d5e4814-3128-4082-aca2-5bbfa715b2e4\") " pod="openshift-multus/multus-additional-cni-plugins-lzrhd"
Sep 11 21:52:29.355267 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:29.353612    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:29.355267 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353622    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5d5e4814-3128-4082-aca2-5bbfa715b2e4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lzrhd\" (UID: \"5d5e4814-3128-4082-aca2-5bbfa715b2e4\") " pod="openshift-multus/multus-additional-cni-plugins-lzrhd"
Sep 11 21:52:29.355267 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353630    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5d8e96d2-170d-48ab-bf18-e43b69834d56-host-run-k8s-cni-cncf-io\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.355267 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353664    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cb2a378c-7071-4cb8-9588-5993651d056f-device-dir\") pod \"aws-ebs-csi-driver-node-bndqs\" (UID: \"cb2a378c-7071-4cb8-9588-5993651d056f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs"
Sep 11 21:52:29.355267 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353673    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d5e4814-3128-4082-aca2-5bbfa715b2e4-cnibin\") pod \"multus-additional-cni-plugins-lzrhd\" (UID: \"5d5e4814-3128-4082-aca2-5bbfa715b2e4\") " pod="openshift-multus/multus-additional-cni-plugins-lzrhd"
Sep 11 21:52:29.355267 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:29.353693    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05523d8a-2c9a-47f3-ad9d-4c508546cd42-original-pull-secret podName:05523d8a-2c9a-47f3-ad9d-4c508546cd42 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:29.853675961 +0000 UTC m=+3.028752327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05523d8a-2c9a-47f3-ad9d-4c508546cd42-original-pull-secret") pod "global-pull-secret-syncer-fc5j4" (UID: "05523d8a-2c9a-47f3-ad9d-4c508546cd42") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:29.355267 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353714    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e0307d0-2de3-46fe-a794-43536ce8eb60-host-slash\") pod \"iptables-alerter-q4cwb\" (UID: \"8e0307d0-2de3-46fe-a794-43536ce8eb60\") " pod="openshift-network-operator/iptables-alerter-q4cwb"
Sep 11 21:52:29.355267 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353724    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cb2a378c-7071-4cb8-9588-5993651d056f-device-dir\") pod \"aws-ebs-csi-driver-node-bndqs\" (UID: \"cb2a378c-7071-4cb8-9588-5993651d056f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs"
Sep 11 21:52:29.355267 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353740    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-run-systemd\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.355267 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353766    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cb2a378c-7071-4cb8-9588-5993651d056f-registration-dir\") pod \"aws-ebs-csi-driver-node-bndqs\" (UID: \"cb2a378c-7071-4cb8-9588-5993651d056f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs"
Sep 11 21:52:29.355267 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353791    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cb2a378c-7071-4cb8-9588-5993651d056f-sys-fs\") pod \"aws-ebs-csi-driver-node-bndqs\" (UID: \"cb2a378c-7071-4cb8-9588-5993651d056f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs"
Sep 11 21:52:29.355267 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353813    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/87753932-baf9-409d-948e-3d9eb12b1095-host\") pod \"node-ca-4vqqq\" (UID: \"87753932-baf9-409d-948e-3d9eb12b1095\") " pod="openshift-image-registry/node-ca-4vqqq"
Sep 11 21:52:29.355267 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353848    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cb2a378c-7071-4cb8-9588-5993651d056f-registration-dir\") pod \"aws-ebs-csi-driver-node-bndqs\" (UID: \"cb2a378c-7071-4cb8-9588-5993651d056f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs"
Sep 11 21:52:29.355267 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.353892    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cb2a378c-7071-4cb8-9588-5993651d056f-sys-fs\") pod \"aws-ebs-csi-driver-node-bndqs\" (UID: \"cb2a378c-7071-4cb8-9588-5993651d056f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs"
Sep 11 21:52:29.355267 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.354055    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5d5e4814-3128-4082-aca2-5bbfa715b2e4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lzrhd\" (UID: \"5d5e4814-3128-4082-aca2-5bbfa715b2e4\") " pod="openshift-multus/multus-additional-cni-plugins-lzrhd"
Sep 11 21:52:29.366176 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.366150    2587 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 21:52:29.366318 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:29.366241    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:52:29.366318 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:29.366260    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:52:29.366318 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:29.366271    2587 projected.go:196] Error preparing data for projected volume kube-api-access-b4vnl for pod openshift-network-diagnostics/network-check-target-hxztv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:29.366499 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:29.366351    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3752cc39-b860-4328-82d5-d67850cd906c-kube-api-access-b4vnl podName:3752cc39-b860-4328-82d5-d67850cd906c nodeName:}" failed. No retries permitted until 2026-09-11 21:52:29.866322853 +0000 UTC m=+3.041399234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-b4vnl" (UniqueName: "kubernetes.io/projected/3752cc39-b860-4328-82d5-d67850cd906c-kube-api-access-b4vnl") pod "network-check-target-hxztv" (UID: "3752cc39-b860-4328-82d5-d67850cd906c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:29.369981 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.369961    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9tqb\" (UniqueName: \"kubernetes.io/projected/5d5e4814-3128-4082-aca2-5bbfa715b2e4-kube-api-access-j9tqb\") pod \"multus-additional-cni-plugins-lzrhd\" (UID: \"5d5e4814-3128-4082-aca2-5bbfa715b2e4\") " pod="openshift-multus/multus-additional-cni-plugins-lzrhd"
Sep 11 21:52:29.370076 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.370010    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqf2b\" (UniqueName: \"kubernetes.io/projected/50fccc19-46f5-4bdc-8e4d-c728752de971-kube-api-access-gqf2b\") pod \"node-resolver-756n4\" (UID: \"50fccc19-46f5-4bdc-8e4d-c728752de971\") " pod="openshift-dns/node-resolver-756n4"
Sep 11 21:52:29.370076 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.370013    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsnn2\" (UniqueName: \"kubernetes.io/projected/5d8e96d2-170d-48ab-bf18-e43b69834d56-kube-api-access-tsnn2\") pod \"multus-492xh\" (UID: \"5d8e96d2-170d-48ab-bf18-e43b69834d56\") " pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.370076 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.370012    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9fcx\" (UniqueName: \"kubernetes.io/projected/cb2a378c-7071-4cb8-9588-5993651d056f-kube-api-access-t9fcx\") pod \"aws-ebs-csi-driver-node-bndqs\" (UID: \"cb2a378c-7071-4cb8-9588-5993651d056f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs"
Sep 11 21:52:29.370443 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.370424    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-556tv\" (UniqueName: \"kubernetes.io/projected/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-kube-api-access-556tv\") pod \"network-metrics-daemon-n87wl\" (UID: \"92a0df5e-f5d1-4331-9ee2-7ed9916f00fd\") " pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:29.454970 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.454938    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-run-systemd\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.454970 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.454980    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/87753932-baf9-409d-948e-3d9eb12b1095-host\") pod \"node-ca-4vqqq\" (UID: \"87753932-baf9-409d-948e-3d9eb12b1095\") " pod="openshift-image-registry/node-ca-4vqqq"
Sep 11 21:52:29.455198 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455004    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-ovnkube-config\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.455198 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455029    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlwcq\" (UniqueName: \"kubernetes.io/projected/8e0307d0-2de3-46fe-a794-43536ce8eb60-kube-api-access-wlwcq\") pod \"iptables-alerter-q4cwb\" (UID: \"8e0307d0-2de3-46fe-a794-43536ce8eb60\") " pod="openshift-network-operator/iptables-alerter-q4cwb"
Sep 11 21:52:29.455198 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455051    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-host-slash\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.455198 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455051    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/87753932-baf9-409d-948e-3d9eb12b1095-host\") pod \"node-ca-4vqqq\" (UID: \"87753932-baf9-409d-948e-3d9eb12b1095\") " pod="openshift-image-registry/node-ca-4vqqq"
Sep 11 21:52:29.455198 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455072    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-ovn-node-metrics-cert\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.455198 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455053    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-run-systemd\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.455198 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455095    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm4ck\" (UniqueName: \"kubernetes.io/projected/741f47df-f402-494f-a78e-03fb0bc10cac-kube-api-access-rm4ck\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.455198 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455115    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-node-log\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.455198 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455139    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-etc-sysconfig\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.455198 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455133    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-host-slash\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.455198 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455158    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-run\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.455198 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455203    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/87753932-baf9-409d-948e-3d9eb12b1095-serviceca\") pod \"node-ca-4vqqq\" (UID: \"87753932-baf9-409d-948e-3d9eb12b1095\") " pod="openshift-image-registry/node-ca-4vqqq"
Sep 11 21:52:29.455887 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455214    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-run\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.455887 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455231    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb56d2f4-9566-4726-81dd-39cf43dbbc42-agent-certs\") pod \"konnectivity-agent-9j5p9\" (UID: \"fb56d2f4-9566-4726-81dd-39cf43dbbc42\") " pod="kube-system/konnectivity-agent-9j5p9"
Sep 11 21:52:29.455887 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455254    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-systemd-units\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.455887 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455276    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-host-cni-bin\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.455887 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455298    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-etc-systemd\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.455887 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455323    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/741f47df-f402-494f-a78e-03fb0bc10cac-etc-tuned\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.455887 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455356    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-systemd-units\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.455887 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455361    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/741f47df-f402-494f-a78e-03fb0bc10cac-tmp\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.455887 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455398    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-node-log\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.455887 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455432    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-log-socket\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.455887 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455468    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-host-cni-bin\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.455887 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455499    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-log-socket\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.455887 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455550    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-etc-sysconfig\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.455887 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455587    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-var-lib-kubelet\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.455887 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455581    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-etc-systemd\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.455887 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455644    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-var-lib-kubelet\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.455887 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455688    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/87753932-baf9-409d-948e-3d9eb12b1095-serviceca\") pod \"node-ca-4vqqq\" (UID: \"87753932-baf9-409d-948e-3d9eb12b1095\") " pod="openshift-image-registry/node-ca-4vqqq"
Sep 11 21:52:29.455887 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455702    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-etc-modprobe-d\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.456754 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455690    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-ovnkube-config\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.456754 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455739    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-lib-modules\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.456754 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455766    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghbk9\" (UniqueName: \"kubernetes.io/projected/87753932-baf9-409d-948e-3d9eb12b1095-kube-api-access-ghbk9\") pod \"node-ca-4vqqq\" (UID: \"87753932-baf9-409d-948e-3d9eb12b1095\") " pod="openshift-image-registry/node-ca-4vqqq"
Sep 11 21:52:29.456754 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455791    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb56d2f4-9566-4726-81dd-39cf43dbbc42-konnectivity-ca\") pod \"konnectivity-agent-9j5p9\" (UID: \"fb56d2f4-9566-4726-81dd-39cf43dbbc42\") " pod="kube-system/konnectivity-agent-9j5p9"
Sep 11 21:52:29.456754 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455814    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-host-kubelet\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.456754 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455837    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-etc-modprobe-d\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.456754 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455842    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-var-lib-openvswitch\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.456754 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455859    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-lib-modules\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.456754 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455879    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-ovnkube-script-lib\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.456754 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455886    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-var-lib-openvswitch\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.456754 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455908    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-etc-openvswitch\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.456754 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455930    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-host-kubelet\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.456754 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455932    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-host-cni-netd\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.456754 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455972    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-host-cni-netd\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.456754 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.455986    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-etc-sysctl-conf\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.456754 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456010    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-run-openvswitch\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.456754 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456049    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-host-run-ovn-kubernetes\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.457298 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456076    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-env-overrides\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.457298 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456101    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-sys\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.457298 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456121    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-host\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.457298 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456143    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8e0307d0-2de3-46fe-a794-43536ce8eb60-iptables-alerter-script\") pod \"iptables-alerter-q4cwb\" (UID: \"8e0307d0-2de3-46fe-a794-43536ce8eb60\") " pod="openshift-network-operator/iptables-alerter-q4cwb"
Sep 11 21:52:29.457298 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456168    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vf5p\" (UniqueName: \"kubernetes.io/projected/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-kube-api-access-2vf5p\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.457298 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456193    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-etc-kubernetes\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.457298 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456215    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-run-ovn\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.457298 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456248    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-host-run-netns\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.457298 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456274    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.457298 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456316    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-etc-sysctl-d\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.457298 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456379    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e0307d0-2de3-46fe-a794-43536ce8eb60-host-slash\") pod \"iptables-alerter-q4cwb\" (UID: \"8e0307d0-2de3-46fe-a794-43536ce8eb60\") " pod="openshift-network-operator/iptables-alerter-q4cwb"
Sep 11 21:52:29.457298 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456405    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb56d2f4-9566-4726-81dd-39cf43dbbc42-konnectivity-ca\") pod \"konnectivity-agent-9j5p9\" (UID: \"fb56d2f4-9566-4726-81dd-39cf43dbbc42\") " pod="kube-system/konnectivity-agent-9j5p9"
Sep 11 21:52:29.457298 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456455    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-etc-openvswitch\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.457298 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456466    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e0307d0-2de3-46fe-a794-43536ce8eb60-host-slash\") pod \"iptables-alerter-q4cwb\" (UID: \"8e0307d0-2de3-46fe-a794-43536ce8eb60\") " pod="openshift-network-operator/iptables-alerter-q4cwb"
Sep 11 21:52:29.457298 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456473    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-host-run-netns\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.457298 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456483    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-ovnkube-script-lib\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.457298 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456518    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-etc-kubernetes\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.457932 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456545    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-run-ovn\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.457932 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456610    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-host-run-ovn-kubernetes\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.457932 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456622    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-etc-sysctl-d\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.457932 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456648    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-run-openvswitch\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.457932 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456664    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-sys\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.457932 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456674    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-etc-sysctl-conf\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.457932 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456707    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/741f47df-f402-494f-a78e-03fb0bc10cac-host\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.457932 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456716    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.457932 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.456912    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8e0307d0-2de3-46fe-a794-43536ce8eb60-iptables-alerter-script\") pod \"iptables-alerter-q4cwb\" (UID: \"8e0307d0-2de3-46fe-a794-43536ce8eb60\") " pod="openshift-network-operator/iptables-alerter-q4cwb"
Sep 11 21:52:29.457932 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.457054    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-env-overrides\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.457932 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.457807    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-ovn-node-metrics-cert\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.458342 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.458107    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/741f47df-f402-494f-a78e-03fb0bc10cac-tmp\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.458532 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.458514    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb56d2f4-9566-4726-81dd-39cf43dbbc42-agent-certs\") pod \"konnectivity-agent-9j5p9\" (UID: \"fb56d2f4-9566-4726-81dd-39cf43dbbc42\") " pod="kube-system/konnectivity-agent-9j5p9"
Sep 11 21:52:29.458567 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.458514    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/741f47df-f402-494f-a78e-03fb0bc10cac-etc-tuned\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.463220 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.463197    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm4ck\" (UniqueName: \"kubernetes.io/projected/741f47df-f402-494f-a78e-03fb0bc10cac-kube-api-access-rm4ck\") pod \"tuned-zs6pz\" (UID: \"741f47df-f402-494f-a78e-03fb0bc10cac\") " pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.463821 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.463804    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghbk9\" (UniqueName: \"kubernetes.io/projected/87753932-baf9-409d-948e-3d9eb12b1095-kube-api-access-ghbk9\") pod \"node-ca-4vqqq\" (UID: \"87753932-baf9-409d-948e-3d9eb12b1095\") " pod="openshift-image-registry/node-ca-4vqqq"
Sep 11 21:52:29.463893 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.463873    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlwcq\" (UniqueName: \"kubernetes.io/projected/8e0307d0-2de3-46fe-a794-43536ce8eb60-kube-api-access-wlwcq\") pod \"iptables-alerter-q4cwb\" (UID: \"8e0307d0-2de3-46fe-a794-43536ce8eb60\") " pod="openshift-network-operator/iptables-alerter-q4cwb"
Sep 11 21:52:29.464090 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.464069    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vf5p\" (UniqueName: \"kubernetes.io/projected/7ab8a820-c9ec-415c-bfaa-3c9744ac040a-kube-api-access-2vf5p\") pod \"ovnkube-node-lw6qf\" (UID: \"7ab8a820-c9ec-415c-bfaa-3c9744ac040a\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.547728 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.547635    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs"
Sep 11 21:52:29.554588 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.554561    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-756n4"
Sep 11 21:52:29.562295 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.562273    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-492xh"
Sep 11 21:52:29.567042 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.567025    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lzrhd"
Sep 11 21:52:29.573715 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.573695    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9j5p9"
Sep 11 21:52:29.580528 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.580500    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:29.587361 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.587342    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zs6pz"
Sep 11 21:52:29.588012 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.587989    2587 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 21:52:29.595338 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.595313    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4vqqq"
Sep 11 21:52:29.600906 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.600885    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q4cwb"
Sep 11 21:52:29.859288 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.859218    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-metrics-certs\") pod \"network-metrics-daemon-n87wl\" (UID: \"92a0df5e-f5d1-4331-9ee2-7ed9916f00fd\") " pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:29.859288 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.859255    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05523d8a-2c9a-47f3-ad9d-4c508546cd42-original-pull-secret\") pod \"global-pull-secret-syncer-fc5j4\" (UID: \"05523d8a-2c9a-47f3-ad9d-4c508546cd42\") " pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:29.859479 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:29.859370    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:29.859479 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:29.859373    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:29.859479 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:29.859418    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05523d8a-2c9a-47f3-ad9d-4c508546cd42-original-pull-secret podName:05523d8a-2c9a-47f3-ad9d-4c508546cd42 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:30.859404983 +0000 UTC m=+4.034481337 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05523d8a-2c9a-47f3-ad9d-4c508546cd42-original-pull-secret") pod "global-pull-secret-syncer-fc5j4" (UID: "05523d8a-2c9a-47f3-ad9d-4c508546cd42") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:29.859479 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:29.859432    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-metrics-certs podName:92a0df5e-f5d1-4331-9ee2-7ed9916f00fd nodeName:}" failed. No retries permitted until 2026-09-11 21:52:30.859426497 +0000 UTC m=+4.034502848 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-metrics-certs") pod "network-metrics-daemon-n87wl" (UID: "92a0df5e-f5d1-4331-9ee2-7ed9916f00fd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:29.959905 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:29.959877    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4vnl\" (UniqueName: \"kubernetes.io/projected/3752cc39-b860-4328-82d5-d67850cd906c-kube-api-access-b4vnl\") pod \"network-check-target-hxztv\" (UID: \"3752cc39-b860-4328-82d5-d67850cd906c\") " pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:52:29.960057 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:29.960029    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:52:29.960057 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:29.960045    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:52:29.960057 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:29.960054    2587 projected.go:196] Error preparing data for projected volume kube-api-access-b4vnl for pod openshift-network-diagnostics/network-check-target-hxztv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:29.960151 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:29.960103    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3752cc39-b860-4328-82d5-d67850cd906c-kube-api-access-b4vnl podName:3752cc39-b860-4328-82d5-d67850cd906c nodeName:}" failed. No retries permitted until 2026-09-11 21:52:30.960089946 +0000 UTC m=+4.135166298 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-b4vnl" (UniqueName: "kubernetes.io/projected/3752cc39-b860-4328-82d5-d67850cd906c-kube-api-access-b4vnl") pod "network-check-target-hxztv" (UID: "3752cc39-b860-4328-82d5-d67850cd906c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:30.186370 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:30.186343    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab8a820_c9ec_415c_bfaa_3c9744ac040a.slice/crio-16f24781a42dab96de7174194884cb0cd4228c7669f2a506d6a91a97344d2067 WatchSource:0}: Error finding container 16f24781a42dab96de7174194884cb0cd4228c7669f2a506d6a91a97344d2067: Status 404 returned error can't find the container with id 16f24781a42dab96de7174194884cb0cd4228c7669f2a506d6a91a97344d2067
Sep 11 21:52:30.188057 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:30.188035    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87753932_baf9_409d_948e_3d9eb12b1095.slice/crio-43828229edf92dbc94658da3a188ef322a11fd7e0d29a4ed2b27c0ab2c25e097 WatchSource:0}: Error finding container 43828229edf92dbc94658da3a188ef322a11fd7e0d29a4ed2b27c0ab2c25e097: Status 404 returned error can't find the container with id 43828229edf92dbc94658da3a188ef322a11fd7e0d29a4ed2b27c0ab2c25e097
Sep 11 21:52:30.188801 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:30.188762    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod741f47df_f402_494f_a78e_03fb0bc10cac.slice/crio-19fc6b9231bad4b8e7563ae101a1cffe703c19a0bc10d14302b350e56fe39d9c WatchSource:0}: Error finding container 19fc6b9231bad4b8e7563ae101a1cffe703c19a0bc10d14302b350e56fe39d9c: Status 404 returned error can't find the container with id 19fc6b9231bad4b8e7563ae101a1cffe703c19a0bc10d14302b350e56fe39d9c
Sep 11 21:52:30.191821 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:30.191790    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5e4814_3128_4082_aca2_5bbfa715b2e4.slice/crio-ac8e22c6f1a5582028f640d3a0c0eb64c23ce63e10ffd6cbe4c2061d60a92c88 WatchSource:0}: Error finding container ac8e22c6f1a5582028f640d3a0c0eb64c23ce63e10ffd6cbe4c2061d60a92c88: Status 404 returned error can't find the container with id ac8e22c6f1a5582028f640d3a0c0eb64c23ce63e10ffd6cbe4c2061d60a92c88
Sep 11 21:52:30.192796 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:30.192684    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e0307d0_2de3_46fe_a794_43536ce8eb60.slice/crio-097625fe7fc55cf3b7d4c473e499d4e871ef604d9280934533ccd66d58e273d3 WatchSource:0}: Error finding container 097625fe7fc55cf3b7d4c473e499d4e871ef604d9280934533ccd66d58e273d3: Status 404 returned error can't find the container with id 097625fe7fc55cf3b7d4c473e499d4e871ef604d9280934533ccd66d58e273d3
Sep 11 21:52:30.194104 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:30.194079    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50fccc19_46f5_4bdc_8e4d_c728752de971.slice/crio-049f36d471f5032de236224ad21007418ff0b5f580444f30c4389f696b80ec8d WatchSource:0}: Error finding container 049f36d471f5032de236224ad21007418ff0b5f580444f30c4389f696b80ec8d: Status 404 returned error can't find the container with id 049f36d471f5032de236224ad21007418ff0b5f580444f30c4389f696b80ec8d
Sep 11 21:52:30.195157 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:30.195132    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb2a378c_7071_4cb8_9588_5993651d056f.slice/crio-40b6407706ec2a465ae095155b5ce75d63f47d34f81c2549aee4ee4ece08b077 WatchSource:0}: Error finding container 40b6407706ec2a465ae095155b5ce75d63f47d34f81c2549aee4ee4ece08b077: Status 404 returned error can't find the container with id 40b6407706ec2a465ae095155b5ce75d63f47d34f81c2549aee4ee4ece08b077
Sep 11 21:52:30.195754 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:30.195728    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d8e96d2_170d_48ab_bf18_e43b69834d56.slice/crio-789374f8a38c939f74793ebc548bc1cffae6d0b7181f46172bf60cadf7906510 WatchSource:0}: Error finding container 789374f8a38c939f74793ebc548bc1cffae6d0b7181f46172bf60cadf7906510: Status 404 returned error can't find the container with id 789374f8a38c939f74793ebc548bc1cffae6d0b7181f46172bf60cadf7906510
Sep 11 21:52:30.198373 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:52:30.197277    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb56d2f4_9566_4726_81dd_39cf43dbbc42.slice/crio-3faeaad1d278028be412741d9a8e927d242af2bf275b26cd37c534473223c45e WatchSource:0}: Error finding container 3faeaad1d278028be412741d9a8e927d242af2bf275b26cd37c534473223c45e: Status 404 returned error can't find the container with id 3faeaad1d278028be412741d9a8e927d242af2bf275b26cd37c534473223c45e
Sep 11 21:52:30.288047 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:30.287984    2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 21:47:29 +0000 UTC" deadline="2028-06-16 12:05:10.795665038 +0000 UTC"
Sep 11 21:52:30.288047 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:30.288048    2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15446h12m40.507625965s"
Sep 11 21:52:30.330272 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:30.330234    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:30.330450 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:30.330398    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc5j4" podUID="05523d8a-2c9a-47f3-ad9d-4c508546cd42"
Sep 11 21:52:30.338775 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:30.338736    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-492xh" event={"ID":"5d8e96d2-170d-48ab-bf18-e43b69834d56","Type":"ContainerStarted","Data":"789374f8a38c939f74793ebc548bc1cffae6d0b7181f46172bf60cadf7906510"}
Sep 11 21:52:30.339938 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:30.339906    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs" event={"ID":"cb2a378c-7071-4cb8-9588-5993651d056f","Type":"ContainerStarted","Data":"40b6407706ec2a465ae095155b5ce75d63f47d34f81c2549aee4ee4ece08b077"}
Sep 11 21:52:30.341084 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:30.341061    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q4cwb" event={"ID":"8e0307d0-2de3-46fe-a794-43536ce8eb60","Type":"ContainerStarted","Data":"097625fe7fc55cf3b7d4c473e499d4e871ef604d9280934533ccd66d58e273d3"}
Sep 11 21:52:30.342165 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:30.342134    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzrhd" event={"ID":"5d5e4814-3128-4082-aca2-5bbfa715b2e4","Type":"ContainerStarted","Data":"ac8e22c6f1a5582028f640d3a0c0eb64c23ce63e10ffd6cbe4c2061d60a92c88"}
Sep 11 21:52:30.343286 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:30.343261    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9j5p9" event={"ID":"fb56d2f4-9566-4726-81dd-39cf43dbbc42","Type":"ContainerStarted","Data":"3faeaad1d278028be412741d9a8e927d242af2bf275b26cd37c534473223c45e"}
Sep 11 21:52:30.344376 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:30.344357    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-756n4" event={"ID":"50fccc19-46f5-4bdc-8e4d-c728752de971","Type":"ContainerStarted","Data":"049f36d471f5032de236224ad21007418ff0b5f580444f30c4389f696b80ec8d"}
Sep 11 21:52:30.345510 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:30.345491    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zs6pz" event={"ID":"741f47df-f402-494f-a78e-03fb0bc10cac","Type":"ContainerStarted","Data":"19fc6b9231bad4b8e7563ae101a1cffe703c19a0bc10d14302b350e56fe39d9c"}
Sep 11 21:52:30.346623 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:30.346601    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4vqqq" event={"ID":"87753932-baf9-409d-948e-3d9eb12b1095","Type":"ContainerStarted","Data":"43828229edf92dbc94658da3a188ef322a11fd7e0d29a4ed2b27c0ab2c25e097"}
Sep 11 21:52:30.347975 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:30.347955    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf" event={"ID":"7ab8a820-c9ec-415c-bfaa-3c9744ac040a","Type":"ContainerStarted","Data":"16f24781a42dab96de7174194884cb0cd4228c7669f2a506d6a91a97344d2067"}
Sep 11 21:52:30.349952 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:30.349931    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-88.ec2.internal" event={"ID":"d56814d1b26653aa054a4eb19dfe6290","Type":"ContainerStarted","Data":"b983a50121b9d7eb5ce4cf16887b04add57d76bbe22b1f7ba97cbd6ce0345190"}
Sep 11 21:52:30.365281 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:30.365242    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-88.ec2.internal" podStartSLOduration=2.365228666 podStartE2EDuration="2.365228666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:30.364972265 +0000 UTC m=+3.540048632" watchObservedRunningTime="2026-09-11 21:52:30.365228666 +0000 UTC m=+3.540305036"
Sep 11 21:52:30.868836 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:30.868743    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05523d8a-2c9a-47f3-ad9d-4c508546cd42-original-pull-secret\") pod \"global-pull-secret-syncer-fc5j4\" (UID: \"05523d8a-2c9a-47f3-ad9d-4c508546cd42\") " pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:30.868996 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:30.868845    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-metrics-certs\") pod \"network-metrics-daemon-n87wl\" (UID: \"92a0df5e-f5d1-4331-9ee2-7ed9916f00fd\") " pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:30.868996 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:30.868974    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:30.869121 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:30.869036    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-metrics-certs podName:92a0df5e-f5d1-4331-9ee2-7ed9916f00fd nodeName:}" failed. No retries permitted until 2026-09-11 21:52:32.869017258 +0000 UTC m=+6.044093620 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-metrics-certs") pod "network-metrics-daemon-n87wl" (UID: "92a0df5e-f5d1-4331-9ee2-7ed9916f00fd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:30.869121 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:30.869102    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:30.869234 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:30.869134    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05523d8a-2c9a-47f3-ad9d-4c508546cd42-original-pull-secret podName:05523d8a-2c9a-47f3-ad9d-4c508546cd42 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:32.869122668 +0000 UTC m=+6.044199022 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05523d8a-2c9a-47f3-ad9d-4c508546cd42-original-pull-secret") pod "global-pull-secret-syncer-fc5j4" (UID: "05523d8a-2c9a-47f3-ad9d-4c508546cd42") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:30.969380 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:30.969339    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4vnl\" (UniqueName: \"kubernetes.io/projected/3752cc39-b860-4328-82d5-d67850cd906c-kube-api-access-b4vnl\") pod \"network-check-target-hxztv\" (UID: \"3752cc39-b860-4328-82d5-d67850cd906c\") " pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:52:30.969577 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:30.969497    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:52:30.969577 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:30.969523    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:52:30.969577 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:30.969538    2587 projected.go:196] Error preparing data for projected volume kube-api-access-b4vnl for pod openshift-network-diagnostics/network-check-target-hxztv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:30.969722 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:30.969599    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3752cc39-b860-4328-82d5-d67850cd906c-kube-api-access-b4vnl podName:3752cc39-b860-4328-82d5-d67850cd906c nodeName:}" failed. No retries permitted until 2026-09-11 21:52:32.969580759 +0000 UTC m=+6.144657111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-b4vnl" (UniqueName: "kubernetes.io/projected/3752cc39-b860-4328-82d5-d67850cd906c-kube-api-access-b4vnl") pod "network-check-target-hxztv" (UID: "3752cc39-b860-4328-82d5-d67850cd906c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:31.288553 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:31.288485    2587 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 21:47:29 +0000 UTC" deadline="2028-03-24 18:45:28.345942784 +0000 UTC"
Sep 11 21:52:31.288553 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:31.288551    2587 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13436h52m57.057396542s"
Sep 11 21:52:31.333494 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:31.332942    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:31.333494 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:31.333097    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n87wl" podUID="92a0df5e-f5d1-4331-9ee2-7ed9916f00fd"
Sep 11 21:52:31.333494 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:31.333142    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:52:31.333494 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:31.333265    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxztv" podUID="3752cc39-b860-4328-82d5-d67850cd906c"
Sep 11 21:52:31.373677 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:31.373620    2587 generic.go:356] "Generic (PLEG): container finished" podID="1f90deb2b4826b854b012a456c786bb2" containerID="215325b16f3110ccd4869883734fd520eb53306185ec679526124486c586683a" exitCode=0
Sep 11 21:52:31.374684 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:31.374654    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal" event={"ID":"1f90deb2b4826b854b012a456c786bb2","Type":"ContainerDied","Data":"215325b16f3110ccd4869883734fd520eb53306185ec679526124486c586683a"}
Sep 11 21:52:32.329796 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:32.329510    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:32.330245 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:32.329862    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc5j4" podUID="05523d8a-2c9a-47f3-ad9d-4c508546cd42"
Sep 11 21:52:32.397168 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:32.397130    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal" event={"ID":"1f90deb2b4826b854b012a456c786bb2","Type":"ContainerStarted","Data":"3972b486075a5c90673e38044b305173291eaf8e3dd5a47f112c856e3cf25031"}
Sep 11 21:52:32.416484 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:32.416412    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-88.ec2.internal" podStartSLOduration=4.416390377 podStartE2EDuration="4.416390377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:32.415719933 +0000 UTC m=+5.590796325" watchObservedRunningTime="2026-09-11 21:52:32.416390377 +0000 UTC m=+5.591466749"
Sep 11 21:52:32.890960 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:32.890124    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-metrics-certs\") pod \"network-metrics-daemon-n87wl\" (UID: \"92a0df5e-f5d1-4331-9ee2-7ed9916f00fd\") " pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:32.890960 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:32.890189    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05523d8a-2c9a-47f3-ad9d-4c508546cd42-original-pull-secret\") pod \"global-pull-secret-syncer-fc5j4\" (UID: \"05523d8a-2c9a-47f3-ad9d-4c508546cd42\") " pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:32.890960 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:32.890376    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:32.890960 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:32.890444    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05523d8a-2c9a-47f3-ad9d-4c508546cd42-original-pull-secret podName:05523d8a-2c9a-47f3-ad9d-4c508546cd42 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:36.890425729 +0000 UTC m=+10.065502094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05523d8a-2c9a-47f3-ad9d-4c508546cd42-original-pull-secret") pod "global-pull-secret-syncer-fc5j4" (UID: "05523d8a-2c9a-47f3-ad9d-4c508546cd42") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:32.890960 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:32.890836    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:32.890960 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:32.890889    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-metrics-certs podName:92a0df5e-f5d1-4331-9ee2-7ed9916f00fd nodeName:}" failed. No retries permitted until 2026-09-11 21:52:36.890874501 +0000 UTC m=+10.065950856 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-metrics-certs") pod "network-metrics-daemon-n87wl" (UID: "92a0df5e-f5d1-4331-9ee2-7ed9916f00fd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:32.991565 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:32.990919    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4vnl\" (UniqueName: \"kubernetes.io/projected/3752cc39-b860-4328-82d5-d67850cd906c-kube-api-access-b4vnl\") pod \"network-check-target-hxztv\" (UID: \"3752cc39-b860-4328-82d5-d67850cd906c\") " pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:52:32.991565 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:32.991091    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:52:32.991565 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:32.991110    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:52:32.991565 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:32.991123    2587 projected.go:196] Error preparing data for projected volume kube-api-access-b4vnl for pod openshift-network-diagnostics/network-check-target-hxztv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:32.991565 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:32.991180    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3752cc39-b860-4328-82d5-d67850cd906c-kube-api-access-b4vnl podName:3752cc39-b860-4328-82d5-d67850cd906c nodeName:}" failed. No retries permitted until 2026-09-11 21:52:36.991161241 +0000 UTC m=+10.166237593 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-b4vnl" (UniqueName: "kubernetes.io/projected/3752cc39-b860-4328-82d5-d67850cd906c-kube-api-access-b4vnl") pod "network-check-target-hxztv" (UID: "3752cc39-b860-4328-82d5-d67850cd906c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:33.330918 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:33.330417    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:52:33.330918 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:33.330541    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxztv" podUID="3752cc39-b860-4328-82d5-d67850cd906c"
Sep 11 21:52:33.331458 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:33.331031    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:33.331458 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:33.331150    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n87wl" podUID="92a0df5e-f5d1-4331-9ee2-7ed9916f00fd"
Sep 11 21:52:34.330041 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:34.330000    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:34.330213 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:34.330142    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc5j4" podUID="05523d8a-2c9a-47f3-ad9d-4c508546cd42"
Sep 11 21:52:35.330221 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:35.330090    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:52:35.330221 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:35.330107    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:35.330812 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:35.330225    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxztv" podUID="3752cc39-b860-4328-82d5-d67850cd906c"
Sep 11 21:52:35.330812 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:35.330663    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n87wl" podUID="92a0df5e-f5d1-4331-9ee2-7ed9916f00fd"
Sep 11 21:52:36.329579 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:36.329541    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:36.329735 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:36.329679    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc5j4" podUID="05523d8a-2c9a-47f3-ad9d-4c508546cd42"
Sep 11 21:52:36.926246 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:36.925998    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-metrics-certs\") pod \"network-metrics-daemon-n87wl\" (UID: \"92a0df5e-f5d1-4331-9ee2-7ed9916f00fd\") " pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:36.926246 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:36.926081    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05523d8a-2c9a-47f3-ad9d-4c508546cd42-original-pull-secret\") pod \"global-pull-secret-syncer-fc5j4\" (UID: \"05523d8a-2c9a-47f3-ad9d-4c508546cd42\") " pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:36.926246 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:36.926179    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:36.926246 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:36.926194    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:36.926897 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:36.926260    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-metrics-certs podName:92a0df5e-f5d1-4331-9ee2-7ed9916f00fd nodeName:}" failed. No retries permitted until 2026-09-11 21:52:44.926236759 +0000 UTC m=+18.101313130 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-metrics-certs") pod "network-metrics-daemon-n87wl" (UID: "92a0df5e-f5d1-4331-9ee2-7ed9916f00fd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:36.926897 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:36.926281    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05523d8a-2c9a-47f3-ad9d-4c508546cd42-original-pull-secret podName:05523d8a-2c9a-47f3-ad9d-4c508546cd42 nodeName:}" failed. No retries permitted until 2026-09-11 21:52:44.926271682 +0000 UTC m=+18.101348032 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05523d8a-2c9a-47f3-ad9d-4c508546cd42-original-pull-secret") pod "global-pull-secret-syncer-fc5j4" (UID: "05523d8a-2c9a-47f3-ad9d-4c508546cd42") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:37.026896 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:37.026862    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4vnl\" (UniqueName: \"kubernetes.io/projected/3752cc39-b860-4328-82d5-d67850cd906c-kube-api-access-b4vnl\") pod \"network-check-target-hxztv\" (UID: \"3752cc39-b860-4328-82d5-d67850cd906c\") " pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:52:37.027077 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:37.027048    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:52:37.027077 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:37.027071    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:52:37.027183 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:37.027083    2587 projected.go:196] Error preparing data for projected volume kube-api-access-b4vnl for pod openshift-network-diagnostics/network-check-target-hxztv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:37.027183 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:37.027142    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3752cc39-b860-4328-82d5-d67850cd906c-kube-api-access-b4vnl podName:3752cc39-b860-4328-82d5-d67850cd906c nodeName:}" failed. No retries permitted until 2026-09-11 21:52:45.027124128 +0000 UTC m=+18.202200481 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-b4vnl" (UniqueName: "kubernetes.io/projected/3752cc39-b860-4328-82d5-d67850cd906c-kube-api-access-b4vnl") pod "network-check-target-hxztv" (UID: "3752cc39-b860-4328-82d5-d67850cd906c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:37.331919 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:37.331032    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:52:37.331919 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:37.331156    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxztv" podUID="3752cc39-b860-4328-82d5-d67850cd906c"
Sep 11 21:52:37.331919 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:37.331561    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:37.331919 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:37.331670    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n87wl" podUID="92a0df5e-f5d1-4331-9ee2-7ed9916f00fd"
Sep 11 21:52:38.330253 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:38.330180    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:38.330734 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:38.330322    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc5j4" podUID="05523d8a-2c9a-47f3-ad9d-4c508546cd42"
Sep 11 21:52:39.330213 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:39.330174    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:39.330422 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:39.330311    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n87wl" podUID="92a0df5e-f5d1-4331-9ee2-7ed9916f00fd"
Sep 11 21:52:39.330422 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:39.330371    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:52:39.330797 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:39.330487    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxztv" podUID="3752cc39-b860-4328-82d5-d67850cd906c"
Sep 11 21:52:40.329466 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:40.329439    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:40.329559 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:40.329538    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc5j4" podUID="05523d8a-2c9a-47f3-ad9d-4c508546cd42"
Sep 11 21:52:41.329679 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:41.329645    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:41.330114 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:41.329707    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:52:41.330114 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:41.329816    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n87wl" podUID="92a0df5e-f5d1-4331-9ee2-7ed9916f00fd"
Sep 11 21:52:41.330114 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:41.329931    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxztv" podUID="3752cc39-b860-4328-82d5-d67850cd906c"
Sep 11 21:52:42.329791 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:42.329758    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:42.330175 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:42.329883    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc5j4" podUID="05523d8a-2c9a-47f3-ad9d-4c508546cd42"
Sep 11 21:52:43.330122 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:43.330073    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:52:43.330574 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:43.330227    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxztv" podUID="3752cc39-b860-4328-82d5-d67850cd906c"
Sep 11 21:52:43.330574 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:43.330282    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:43.330574 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:43.330426    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n87wl" podUID="92a0df5e-f5d1-4331-9ee2-7ed9916f00fd"
Sep 11 21:52:44.330231 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:44.330197    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:44.330771 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:44.330312    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc5j4" podUID="05523d8a-2c9a-47f3-ad9d-4c508546cd42"
Sep 11 21:52:44.982136 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:44.982094    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-metrics-certs\") pod \"network-metrics-daemon-n87wl\" (UID: \"92a0df5e-f5d1-4331-9ee2-7ed9916f00fd\") " pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:44.982344 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:44.982157    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05523d8a-2c9a-47f3-ad9d-4c508546cd42-original-pull-secret\") pod \"global-pull-secret-syncer-fc5j4\" (UID: \"05523d8a-2c9a-47f3-ad9d-4c508546cd42\") " pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:44.982344 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:44.982252    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:44.982344 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:44.982253    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:44.982344 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:44.982316    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05523d8a-2c9a-47f3-ad9d-4c508546cd42-original-pull-secret podName:05523d8a-2c9a-47f3-ad9d-4c508546cd42 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:00.982297969 +0000 UTC m=+34.157374341 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05523d8a-2c9a-47f3-ad9d-4c508546cd42-original-pull-secret") pod "global-pull-secret-syncer-fc5j4" (UID: "05523d8a-2c9a-47f3-ad9d-4c508546cd42") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:52:44.982519 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:44.982354    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-metrics-certs podName:92a0df5e-f5d1-4331-9ee2-7ed9916f00fd nodeName:}" failed. No retries permitted until 2026-09-11 21:53:00.982342662 +0000 UTC m=+34.157419027 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-metrics-certs") pod "network-metrics-daemon-n87wl" (UID: "92a0df5e-f5d1-4331-9ee2-7ed9916f00fd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:52:45.083066 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:45.083021    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4vnl\" (UniqueName: \"kubernetes.io/projected/3752cc39-b860-4328-82d5-d67850cd906c-kube-api-access-b4vnl\") pod \"network-check-target-hxztv\" (UID: \"3752cc39-b860-4328-82d5-d67850cd906c\") " pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:52:45.083242 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:45.083179    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:52:45.083242 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:45.083199    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:52:45.083242 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:45.083208    2587 projected.go:196] Error preparing data for projected volume kube-api-access-b4vnl for pod openshift-network-diagnostics/network-check-target-hxztv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:45.083395 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:45.083267    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3752cc39-b860-4328-82d5-d67850cd906c-kube-api-access-b4vnl podName:3752cc39-b860-4328-82d5-d67850cd906c nodeName:}" failed. No retries permitted until 2026-09-11 21:53:01.083251004 +0000 UTC m=+34.258327369 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-b4vnl" (UniqueName: "kubernetes.io/projected/3752cc39-b860-4328-82d5-d67850cd906c-kube-api-access-b4vnl") pod "network-check-target-hxztv" (UID: "3752cc39-b860-4328-82d5-d67850cd906c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:52:45.330419 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:45.330309    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:52:45.330848 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:45.330468    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxztv" podUID="3752cc39-b860-4328-82d5-d67850cd906c"
Sep 11 21:52:45.330848 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:45.330527    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:45.330848 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:45.330656    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n87wl" podUID="92a0df5e-f5d1-4331-9ee2-7ed9916f00fd"
Sep 11 21:52:46.330290 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:46.330247    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:46.330488 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:46.330379    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc5j4" podUID="05523d8a-2c9a-47f3-ad9d-4c508546cd42"
Sep 11 21:52:47.331040 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:47.330997    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:47.331665 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:47.331049    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:52:47.331665 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:47.331147    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n87wl" podUID="92a0df5e-f5d1-4331-9ee2-7ed9916f00fd"
Sep 11 21:52:47.331665 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:47.331272    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxztv" podUID="3752cc39-b860-4328-82d5-d67850cd906c"
Sep 11 21:52:48.330189 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:48.330016    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:48.330283 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:48.330264    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc5j4" podUID="05523d8a-2c9a-47f3-ad9d-4c508546cd42"
Sep 11 21:52:48.429911 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:48.429881    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9j5p9" event={"ID":"fb56d2f4-9566-4726-81dd-39cf43dbbc42","Type":"ContainerStarted","Data":"04ea9161eb94ffca596f2909af92337b436057cdb42b3890c33cb00b696f3cdd"}
Sep 11 21:52:48.432284 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:48.432157    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-756n4" event={"ID":"50fccc19-46f5-4bdc-8e4d-c728752de971","Type":"ContainerStarted","Data":"1aa25b00503693949e22f422211d937dc8ddaa7794edb55e0acdfe5ca8b91b24"}
Sep 11 21:52:48.434489 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:48.434464    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zs6pz" event={"ID":"741f47df-f402-494f-a78e-03fb0bc10cac","Type":"ContainerStarted","Data":"24ef0e490a72f8fec125d391f9f4c3d43682e5b0de01b66cc452301c0b201018"}
Sep 11 21:52:48.436207 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:48.436182    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4vqqq" event={"ID":"87753932-baf9-409d-948e-3d9eb12b1095","Type":"ContainerStarted","Data":"30817c1198d38a9b9f33af88c04dafe425907ed5159ed11ca7abb88b68fa8110"}
Sep 11 21:52:48.438883 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:48.438864    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw6qf_7ab8a820-c9ec-415c-bfaa-3c9744ac040a/ovn-acl-logging/0.log"
Sep 11 21:52:48.439221 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:48.439201    2587 generic.go:356] "Generic (PLEG): container finished" podID="7ab8a820-c9ec-415c-bfaa-3c9744ac040a" containerID="d9399904b17954fb4cf81531b33326c7a136bca2b7270e8b13dd3a231394d102" exitCode=1
Sep 11 21:52:48.439348 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:48.439238    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf" event={"ID":"7ab8a820-c9ec-415c-bfaa-3c9744ac040a","Type":"ContainerStarted","Data":"1250b6a50b6b1430624375f7e8d95654f6deb5d36161690028dd3b135bd76e94"}
Sep 11 21:52:48.439348 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:48.439292    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf" event={"ID":"7ab8a820-c9ec-415c-bfaa-3c9744ac040a","Type":"ContainerStarted","Data":"d2ba1c33833331d361264e281af7659db6b936fc69d545a9f1d08665995bd6a3"}
Sep 11 21:52:48.439348 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:48.439321    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf" event={"ID":"7ab8a820-c9ec-415c-bfaa-3c9744ac040a","Type":"ContainerStarted","Data":"fce344f4e10558383ce066cea27efff579d87d9e7f362b00f5d7e75837d68ba4"}
Sep 11 21:52:48.439575 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:48.439367    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf" event={"ID":"7ab8a820-c9ec-415c-bfaa-3c9744ac040a","Type":"ContainerDied","Data":"d9399904b17954fb4cf81531b33326c7a136bca2b7270e8b13dd3a231394d102"}
Sep 11 21:52:48.439575 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:48.439399    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf" event={"ID":"7ab8a820-c9ec-415c-bfaa-3c9744ac040a","Type":"ContainerStarted","Data":"1f4ac8eea3baf8f07345ee92f14b59cf0621ed4ecd6e683767bc062f5c52e70c"}
Sep 11 21:52:48.441277 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:48.441254    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-492xh" event={"ID":"5d8e96d2-170d-48ab-bf18-e43b69834d56","Type":"ContainerStarted","Data":"a240a9d3d112a508d8d9b2c7c5e970d7661a6197a2ff13bb4d3b47a78ae0d64b"}
Sep 11 21:52:48.443163 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:48.443144    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs" event={"ID":"cb2a378c-7071-4cb8-9588-5993651d056f","Type":"ContainerStarted","Data":"0f1cc13ae617add55106aa7cc6c5bbe1259c34dc485a91314d6421e10588265e"}
Sep 11 21:52:48.445057 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:48.445033    2587 generic.go:356] "Generic (PLEG): container finished" podID="5d5e4814-3128-4082-aca2-5bbfa715b2e4" containerID="20ea4b883fbc5b19335e285f779aade331c0786ca9aac0491c58d8bb58915915" exitCode=0
Sep 11 21:52:48.445135 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:48.445078    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzrhd" event={"ID":"5d5e4814-3128-4082-aca2-5bbfa715b2e4","Type":"ContainerDied","Data":"20ea4b883fbc5b19335e285f779aade331c0786ca9aac0491c58d8bb58915915"}
Sep 11 21:52:48.486714 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:48.486603    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9j5p9" podStartSLOduration=3.965529076 podStartE2EDuration="21.486585059s" totalImagesPullingTime="17.521055983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:48.448840838 +0000 UTC m=+21.623917200" watchObservedRunningTime="2026-09-11 21:52:48.486585059 +0000 UTC m=+21.661661430"
Sep 11 21:52:48.514019 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:48.513231    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-756n4" podStartSLOduration=3.962781473 podStartE2EDuration="21.513212594s" totalImagesPullingTime="17.550431121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:48.512829626 +0000 UTC m=+21.687906012" watchObservedRunningTime="2026-09-11 21:52:48.513212594 +0000 UTC m=+21.688288965"
Sep 11 21:52:48.514019 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:48.513758    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4vqqq" podStartSLOduration=3.9832238650000003 podStartE2EDuration="21.513744999s" totalImagesPullingTime="17.530521134s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:48.48636972 +0000 UTC m=+21.661446112" watchObservedRunningTime="2026-09-11 21:52:48.513744999 +0000 UTC m=+21.688821372"
Sep 11 21:52:48.534026 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:48.533975    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-492xh" podStartSLOduration=3.760908497 podStartE2EDuration="21.533954915s" totalImagesPullingTime="17.773046418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:48.533477072 +0000 UTC m=+21.708553462" watchObservedRunningTime="2026-09-11 21:52:48.533954915 +0000 UTC m=+21.709031286"
Sep 11 21:52:48.571470 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:48.571413    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zs6pz" podStartSLOduration=4.04221608 podStartE2EDuration="21.571397628s" totalImagesPullingTime="17.529181548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:48.57086928 +0000 UTC m=+21.745945661" watchObservedRunningTime="2026-09-11 21:52:48.571397628 +0000 UTC m=+21.746473992"
Sep 11 21:52:49.281408 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:49.281357    2587 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 21:52:49.329625 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:49.329590    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:52:49.329776 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:49.329695    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxztv" podUID="3752cc39-b860-4328-82d5-d67850cd906c"
Sep 11 21:52:49.329776 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:49.329714    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:49.329899 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:49.329822    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n87wl" podUID="92a0df5e-f5d1-4331-9ee2-7ed9916f00fd"
Sep 11 21:52:49.418571 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:49.418430    2587 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T21:52:49.281391265Z","UUID":"a60e07ff-689c-4614-9347-a20239bdfb15","Handler":null,"Name":"","Endpoint":""}
Sep 11 21:52:49.420241 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:49.420219    2587 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 21:52:49.420397 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:49.420250    2587 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 21:52:49.451212 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:49.451173    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw6qf_7ab8a820-c9ec-415c-bfaa-3c9744ac040a/ovn-acl-logging/0.log"
Sep 11 21:52:49.451840 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:49.451696    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf" event={"ID":"7ab8a820-c9ec-415c-bfaa-3c9744ac040a","Type":"ContainerStarted","Data":"59ebc650bb9913f4f7230ebfba296e0dddfcf5ef32144468de19f5036cef11c4"}
Sep 11 21:52:49.453718 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:49.453684    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs" event={"ID":"cb2a378c-7071-4cb8-9588-5993651d056f","Type":"ContainerStarted","Data":"11f1b8fac54ccf3fdcd799bf8b1507eff7d79bbfe291daeb312ef26eb4f4ba43"}
Sep 11 21:52:49.455553 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:49.455526    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q4cwb" event={"ID":"8e0307d0-2de3-46fe-a794-43536ce8eb60","Type":"ContainerStarted","Data":"c1039b1b3f51e472b2449ec6a07266bc26300d726d7ea1acb3ef8c3bbd11cddd"}
Sep 11 21:52:49.474730 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:49.474678    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-q4cwb" podStartSLOduration=4.949338805 podStartE2EDuration="22.474660254s" totalImagesPullingTime="17.525321449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:49.474638501 +0000 UTC m=+22.649714892" watchObservedRunningTime="2026-09-11 21:52:49.474660254 +0000 UTC m=+22.649736626"
Sep 11 21:52:50.329445 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:50.329413    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:50.329641 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:50.329546    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc5j4" podUID="05523d8a-2c9a-47f3-ad9d-4c508546cd42"
Sep 11 21:52:50.460087 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:50.459998    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs" event={"ID":"cb2a378c-7071-4cb8-9588-5993651d056f","Type":"ContainerStarted","Data":"ae01b21b76963d203a698d214656bbe2d2db7cee64e260d88b6db61d54195286"}
Sep 11 21:52:50.479892 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:50.479845    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bndqs" podStartSLOduration=3.913541812 podStartE2EDuration="23.479830556s" totalImagesPullingTime="19.566288744s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:50.479606634 +0000 UTC m=+23.654683030" watchObservedRunningTime="2026-09-11 21:52:50.479830556 +0000 UTC m=+23.654906926"
Sep 11 21:52:51.106671 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:51.106627    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9j5p9"
Sep 11 21:52:51.107221 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:51.107194    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9j5p9"
Sep 11 21:52:51.330229 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:51.330194    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:52:51.330422 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:51.330320    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxztv" podUID="3752cc39-b860-4328-82d5-d67850cd906c"
Sep 11 21:52:51.330422 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:51.330357    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:51.330528 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:51.330462    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n87wl" podUID="92a0df5e-f5d1-4331-9ee2-7ed9916f00fd"
Sep 11 21:52:51.466403 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:51.466378    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw6qf_7ab8a820-c9ec-415c-bfaa-3c9744ac040a/ovn-acl-logging/0.log"
Sep 11 21:52:51.467020 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:51.466834    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf" event={"ID":"7ab8a820-c9ec-415c-bfaa-3c9744ac040a","Type":"ContainerStarted","Data":"f06b7c7163b17e12f731fb0cf67aaa3b39b7fa7959665257dbc7ffe2e0d755c6"}
Sep 11 21:52:51.467089 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:51.467053    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9j5p9"
Sep 11 21:52:51.467617 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:51.467597    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9j5p9"
Sep 11 21:52:52.329973 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:52.329934    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:52.330127 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:52.330045    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc5j4" podUID="05523d8a-2c9a-47f3-ad9d-4c508546cd42"
Sep 11 21:52:53.329982 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:53.329728    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:52:53.330450 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:53.329808    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:53.330450 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:53.330092    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxztv" podUID="3752cc39-b860-4328-82d5-d67850cd906c"
Sep 11 21:52:53.330450 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:53.330144    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n87wl" podUID="92a0df5e-f5d1-4331-9ee2-7ed9916f00fd"
Sep 11 21:52:53.473833 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:53.473790    2587 generic.go:356] "Generic (PLEG): container finished" podID="5d5e4814-3128-4082-aca2-5bbfa715b2e4" containerID="bd0998f93a00f4f4889fe62449ed39ffd45e0cf04af80b95c51062bcd2b0d348" exitCode=0
Sep 11 21:52:53.473833 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:53.473827    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzrhd" event={"ID":"5d5e4814-3128-4082-aca2-5bbfa715b2e4","Type":"ContainerDied","Data":"bd0998f93a00f4f4889fe62449ed39ffd45e0cf04af80b95c51062bcd2b0d348"}
Sep 11 21:52:53.477595 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:53.477579    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw6qf_7ab8a820-c9ec-415c-bfaa-3c9744ac040a/ovn-acl-logging/0.log"
Sep 11 21:52:53.477994 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:53.477971    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf" event={"ID":"7ab8a820-c9ec-415c-bfaa-3c9744ac040a","Type":"ContainerStarted","Data":"c7d41cf931f9510e75d5d5688db79ba0a5a241d8893abb1473b21e1fe9b7ace3"}
Sep 11 21:52:53.478307 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:53.478293    2587 scope.go:117] "RemoveContainer" containerID="d9399904b17954fb4cf81531b33326c7a136bca2b7270e8b13dd3a231394d102"
Sep 11 21:52:54.330023 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:54.329993    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:54.330378 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:54.330085    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc5j4" podUID="05523d8a-2c9a-47f3-ad9d-4c508546cd42"
Sep 11 21:52:54.484840 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:54.484819    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw6qf_7ab8a820-c9ec-415c-bfaa-3c9744ac040a/ovn-acl-logging/0.log"
Sep 11 21:52:54.485260 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:54.485231    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf" event={"ID":"7ab8a820-c9ec-415c-bfaa-3c9744ac040a","Type":"ContainerStarted","Data":"c70e647868fde639f0f0efe917e21b0b4f463c8dac1f6d2d9089dcf81ad6b8f8"}
Sep 11 21:52:54.485645 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:54.485626    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:54.485758 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:54.485742    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:54.485817 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:54.485805    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:54.488273 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:54.488245    2587 generic.go:356] "Generic (PLEG): container finished" podID="5d5e4814-3128-4082-aca2-5bbfa715b2e4" containerID="6345d40852e7d74d24b7c829981f71205236e573c4143775fdbadecfb0c64f30" exitCode=0
Sep 11 21:52:54.488385 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:54.488296    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzrhd" event={"ID":"5d5e4814-3128-4082-aca2-5bbfa715b2e4","Type":"ContainerDied","Data":"6345d40852e7d74d24b7c829981f71205236e573c4143775fdbadecfb0c64f30"}
Sep 11 21:52:54.501969 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:54.501945    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:54.502074 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:54.502051    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:52:54.516964 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:54.516908    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf" podStartSLOduration=9.947016093 podStartE2EDuration="27.516890685s" totalImagesPullingTime="17.569874592s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:52:54.513775393 +0000 UTC m=+27.688851755" watchObservedRunningTime="2026-09-11 21:52:54.516890685 +0000 UTC m=+27.691967052"
Sep 11 21:52:54.705838 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:54.705808    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fc5j4"]
Sep 11 21:52:54.706027 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:54.705953    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:54.706096 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:54.706050    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc5j4" podUID="05523d8a-2c9a-47f3-ad9d-4c508546cd42"
Sep 11 21:52:54.706523 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:54.706501    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hxztv"]
Sep 11 21:52:54.706631 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:54.706599    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:52:54.706693 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:54.706674    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxztv" podUID="3752cc39-b860-4328-82d5-d67850cd906c"
Sep 11 21:52:54.707279 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:54.707253    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n87wl"]
Sep 11 21:52:54.707396 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:54.707375    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:54.707493 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:54.707474    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n87wl" podUID="92a0df5e-f5d1-4331-9ee2-7ed9916f00fd"
Sep 11 21:52:55.492817 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:55.492739    2587 generic.go:356] "Generic (PLEG): container finished" podID="5d5e4814-3128-4082-aca2-5bbfa715b2e4" containerID="67761ba3b51bf796cb3c2213ad8da7d45862f9d5379e106aecc5efc9d432303f" exitCode=0
Sep 11 21:52:55.492817 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:55.492805    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzrhd" event={"ID":"5d5e4814-3128-4082-aca2-5bbfa715b2e4","Type":"ContainerDied","Data":"67761ba3b51bf796cb3c2213ad8da7d45862f9d5379e106aecc5efc9d432303f"}
Sep 11 21:52:56.329389 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:56.329351    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:56.329585 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:56.329404    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:56.329585 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:56.329461    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc5j4" podUID="05523d8a-2c9a-47f3-ad9d-4c508546cd42"
Sep 11 21:52:56.329585 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:56.329404    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:52:56.329585 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:56.329509    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n87wl" podUID="92a0df5e-f5d1-4331-9ee2-7ed9916f00fd"
Sep 11 21:52:56.329785 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:56.329589    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxztv" podUID="3752cc39-b860-4328-82d5-d67850cd906c"
Sep 11 21:52:58.330176 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:58.330137    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:52:58.330176 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:58.330170    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:52:58.330839 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:58.330185    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:52:58.330839 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:58.330309    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n87wl" podUID="92a0df5e-f5d1-4331-9ee2-7ed9916f00fd"
Sep 11 21:52:58.330839 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:58.330452    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxztv" podUID="3752cc39-b860-4328-82d5-d67850cd906c"
Sep 11 21:52:58.330839 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:52:58.330606    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc5j4" podUID="05523d8a-2c9a-47f3-ad9d-4c508546cd42"
Sep 11 21:52:59.883825 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:52:59.883761    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-756n4_50fccc19-46f5-4bdc-8e4d-c728752de971/dns-node-resolver/0.log"
Sep 11 21:53:00.329521 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:00.329496    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:53:00.329724 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:00.329620    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:53:00.329724 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:00.329632    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hxztv" podUID="3752cc39-b860-4328-82d5-d67850cd906c"
Sep 11 21:53:00.329836 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:00.329747    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n87wl" podUID="92a0df5e-f5d1-4331-9ee2-7ed9916f00fd"
Sep 11 21:53:00.329836 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:00.329807    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:53:00.329912 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:00.329892    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fc5j4" podUID="05523d8a-2c9a-47f3-ad9d-4c508546cd42"
Sep 11 21:53:00.507508 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:00.507469    2587 generic.go:356] "Generic (PLEG): container finished" podID="5d5e4814-3128-4082-aca2-5bbfa715b2e4" containerID="c01082011d428079d01100404590bf978350d9d4d280565b49dd796b8f28591d" exitCode=0
Sep 11 21:53:00.507686 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:00.507541    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzrhd" event={"ID":"5d5e4814-3128-4082-aca2-5bbfa715b2e4","Type":"ContainerDied","Data":"c01082011d428079d01100404590bf978350d9d4d280565b49dd796b8f28591d"}
Sep 11 21:53:00.851659 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:00.851634    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4vqqq_87753932-baf9-409d-948e-3d9eb12b1095/node-ca/0.log"
Sep 11 21:53:01.008268 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.008232    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-metrics-certs\") pod \"network-metrics-daemon-n87wl\" (UID: \"92a0df5e-f5d1-4331-9ee2-7ed9916f00fd\") " pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:53:01.008770 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.008273    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05523d8a-2c9a-47f3-ad9d-4c508546cd42-original-pull-secret\") pod \"global-pull-secret-syncer-fc5j4\" (UID: \"05523d8a-2c9a-47f3-ad9d-4c508546cd42\") " pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:53:01.008770 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:01.008409    2587 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 21:53:01.008770 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:01.008471    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05523d8a-2c9a-47f3-ad9d-4c508546cd42-original-pull-secret podName:05523d8a-2c9a-47f3-ad9d-4c508546cd42 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:33.008456617 +0000 UTC m=+66.183532978 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/05523d8a-2c9a-47f3-ad9d-4c508546cd42-original-pull-secret") pod "global-pull-secret-syncer-fc5j4" (UID: "05523d8a-2c9a-47f3-ad9d-4c508546cd42") : object "kube-system"/"original-pull-secret" not registered
Sep 11 21:53:01.008770 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:01.008409    2587 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:53:01.008770 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:01.008562    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-metrics-certs podName:92a0df5e-f5d1-4331-9ee2-7ed9916f00fd nodeName:}" failed. No retries permitted until 2026-09-11 21:53:33.008544509 +0000 UTC m=+66.183620862 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-metrics-certs") pod "network-metrics-daemon-n87wl" (UID: "92a0df5e-f5d1-4331-9ee2-7ed9916f00fd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 21:53:01.109283 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.109200    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4vnl\" (UniqueName: \"kubernetes.io/projected/3752cc39-b860-4328-82d5-d67850cd906c-kube-api-access-b4vnl\") pod \"network-check-target-hxztv\" (UID: \"3752cc39-b860-4328-82d5-d67850cd906c\") " pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:53:01.109438 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:01.109355    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 21:53:01.109438 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:01.109372    2587 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 21:53:01.109438 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:01.109381    2587 projected.go:196] Error preparing data for projected volume kube-api-access-b4vnl for pod openshift-network-diagnostics/network-check-target-hxztv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:53:01.109438 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:01.109434    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3752cc39-b860-4328-82d5-d67850cd906c-kube-api-access-b4vnl podName:3752cc39-b860-4328-82d5-d67850cd906c nodeName:}" failed. No retries permitted until 2026-09-11 21:53:33.109418538 +0000 UTC m=+66.284494901 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-b4vnl" (UniqueName: "kubernetes.io/projected/3752cc39-b860-4328-82d5-d67850cd906c-kube-api-access-b4vnl") pod "network-check-target-hxztv" (UID: "3752cc39-b860-4328-82d5-d67850cd906c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 21:53:01.512941 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.512908    2587 generic.go:356] "Generic (PLEG): container finished" podID="5d5e4814-3128-4082-aca2-5bbfa715b2e4" containerID="3edd9277112259beb1b38027b1c86a960d1e6a4de62d79cb09ad349055280838" exitCode=0
Sep 11 21:53:01.513062 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.512982    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzrhd" event={"ID":"5d5e4814-3128-4082-aca2-5bbfa715b2e4","Type":"ContainerDied","Data":"3edd9277112259beb1b38027b1c86a960d1e6a4de62d79cb09ad349055280838"}
Sep 11 21:53:01.634642 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.634616    2587 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-88.ec2.internal" event="NodeReady"
Sep 11 21:53:01.634778 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.634765    2587 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 21:53:01.683271 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.683244    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dnlj8"]
Sep 11 21:53:01.702147 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.702115    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rn9mr"]
Sep 11 21:53:01.702281 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.702230    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dnlj8"
Sep 11 21:53:01.705126 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.705103    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 21:53:01.705270 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.705213    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 21:53:01.705270 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.705255    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8rpn9\""
Sep 11 21:53:01.705402 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.705291    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 21:53:01.724150 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.724088    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dnlj8"]
Sep 11 21:53:01.724150 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.724123    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-c9qjm"]
Sep 11 21:53:01.724960 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.724216    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rn9mr"
Sep 11 21:53:01.727717 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.727691    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8zqrn\""
Sep 11 21:53:01.728941 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.728454    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 21:53:01.728941 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.728758    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 21:53:01.746341 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.746300    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rn9mr"]
Sep 11 21:53:01.746469 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.746353    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c9qjm"]
Sep 11 21:53:01.746469 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.746429    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c9qjm"
Sep 11 21:53:01.750699 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.750671    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 21:53:01.750699 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.750671    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 21:53:01.750876 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.750740    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k6pq6\""
Sep 11 21:53:01.750876 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.750679    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 21:53:01.751891 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.751711    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 21:53:01.813788 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.813748    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlrk6\" (UniqueName: \"kubernetes.io/projected/10261a6e-338b-4c67-99bd-6921a24e051b-kube-api-access-hlrk6\") pod \"dns-default-rn9mr\" (UID: \"10261a6e-338b-4c67-99bd-6921a24e051b\") " pod="openshift-dns/dns-default-rn9mr"
Sep 11 21:53:01.813954 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.813809    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10261a6e-338b-4c67-99bd-6921a24e051b-tmp-dir\") pod \"dns-default-rn9mr\" (UID: \"10261a6e-338b-4c67-99bd-6921a24e051b\") " pod="openshift-dns/dns-default-rn9mr"
Sep 11 21:53:01.813954 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.813834    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10261a6e-338b-4c67-99bd-6921a24e051b-metrics-tls\") pod \"dns-default-rn9mr\" (UID: \"10261a6e-338b-4c67-99bd-6921a24e051b\") " pod="openshift-dns/dns-default-rn9mr"
Sep 11 21:53:01.813954 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.813860    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3cdabfb8-9e10-4fe2-9ba4-4301f2ea9fd0-cert\") pod \"ingress-canary-dnlj8\" (UID: \"3cdabfb8-9e10-4fe2-9ba4-4301f2ea9fd0\") " pod="openshift-ingress-canary/ingress-canary-dnlj8"
Sep 11 21:53:01.813954 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.813874    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2m4h\" (UniqueName: \"kubernetes.io/projected/3cdabfb8-9e10-4fe2-9ba4-4301f2ea9fd0-kube-api-access-q2m4h\") pod \"ingress-canary-dnlj8\" (UID: \"3cdabfb8-9e10-4fe2-9ba4-4301f2ea9fd0\") " pod="openshift-ingress-canary/ingress-canary-dnlj8"
Sep 11 21:53:01.813954 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.813899    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/10261a6e-338b-4c67-99bd-6921a24e051b-config-volume\") pod \"dns-default-rn9mr\" (UID: \"10261a6e-338b-4c67-99bd-6921a24e051b\") " pod="openshift-dns/dns-default-rn9mr"
Sep 11 21:53:01.888292 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.888261    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-6xt8x"]
Sep 11 21:53:01.914157 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.914123    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-6xt8x"]
Sep 11 21:53:01.914315 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.914190    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3cdabfb8-9e10-4fe2-9ba4-4301f2ea9fd0-cert\") pod \"ingress-canary-dnlj8\" (UID: \"3cdabfb8-9e10-4fe2-9ba4-4301f2ea9fd0\") " pod="openshift-ingress-canary/ingress-canary-dnlj8"
Sep 11 21:53:01.914315 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.914215    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-6xt8x"
Sep 11 21:53:01.914315 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.914218    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2m4h\" (UniqueName: \"kubernetes.io/projected/3cdabfb8-9e10-4fe2-9ba4-4301f2ea9fd0-kube-api-access-q2m4h\") pod \"ingress-canary-dnlj8\" (UID: \"3cdabfb8-9e10-4fe2-9ba4-4301f2ea9fd0\") " pod="openshift-ingress-canary/ingress-canary-dnlj8"
Sep 11 21:53:01.914315 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.914242    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-data-volume\") pod \"insights-runtime-extractor-c9qjm\" (UID: \"3fdc3282-48a9-49ae-a5a4-5d60fb93639a\") " pod="openshift-insights/insights-runtime-extractor-c9qjm"
Sep 11 21:53:01.914315 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.914273    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/10261a6e-338b-4c67-99bd-6921a24e051b-config-volume\") pod \"dns-default-rn9mr\" (UID: \"10261a6e-338b-4c67-99bd-6921a24e051b\") " pod="openshift-dns/dns-default-rn9mr"
Sep 11 21:53:01.914315 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.914295    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-crio-socket\") pod \"insights-runtime-extractor-c9qjm\" (UID: \"3fdc3282-48a9-49ae-a5a4-5d60fb93639a\") " pod="openshift-insights/insights-runtime-extractor-c9qjm"
Sep 11 21:53:01.914651 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.914371    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9qjm\" (UID: \"3fdc3282-48a9-49ae-a5a4-5d60fb93639a\") " pod="openshift-insights/insights-runtime-extractor-c9qjm"
Sep 11 21:53:01.914651 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.914403    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlrk6\" (UniqueName: \"kubernetes.io/projected/10261a6e-338b-4c67-99bd-6921a24e051b-kube-api-access-hlrk6\") pod \"dns-default-rn9mr\" (UID: \"10261a6e-338b-4c67-99bd-6921a24e051b\") " pod="openshift-dns/dns-default-rn9mr"
Sep 11 21:53:01.914651 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.914458    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10261a6e-338b-4c67-99bd-6921a24e051b-tmp-dir\") pod \"dns-default-rn9mr\" (UID: \"10261a6e-338b-4c67-99bd-6921a24e051b\") " pod="openshift-dns/dns-default-rn9mr"
Sep 11 21:53:01.914651 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.914483    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10261a6e-338b-4c67-99bd-6921a24e051b-metrics-tls\") pod \"dns-default-rn9mr\" (UID: \"10261a6e-338b-4c67-99bd-6921a24e051b\") " pod="openshift-dns/dns-default-rn9mr"
Sep 11 21:53:01.914651 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.914508    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c9qjm\" (UID: \"3fdc3282-48a9-49ae-a5a4-5d60fb93639a\") " pod="openshift-insights/insights-runtime-extractor-c9qjm"
Sep 11 21:53:01.914651 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.914531    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7g7l\" (UniqueName: \"kubernetes.io/projected/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-kube-api-access-l7g7l\") pod \"insights-runtime-extractor-c9qjm\" (UID: \"3fdc3282-48a9-49ae-a5a4-5d60fb93639a\") " pod="openshift-insights/insights-runtime-extractor-c9qjm"
Sep 11 21:53:01.914932 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:01.914653    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:53:01.914932 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:01.914739    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10261a6e-338b-4c67-99bd-6921a24e051b-metrics-tls podName:10261a6e-338b-4c67-99bd-6921a24e051b nodeName:}" failed. No retries permitted until 2026-09-11 21:53:02.414718381 +0000 UTC m=+35.589794750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10261a6e-338b-4c67-99bd-6921a24e051b-metrics-tls") pod "dns-default-rn9mr" (UID: "10261a6e-338b-4c67-99bd-6921a24e051b") : secret "dns-default-metrics-tls" not found
Sep 11 21:53:01.914932 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.914886    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/10261a6e-338b-4c67-99bd-6921a24e051b-config-volume\") pod \"dns-default-rn9mr\" (UID: \"10261a6e-338b-4c67-99bd-6921a24e051b\") " pod="openshift-dns/dns-default-rn9mr"
Sep 11 21:53:01.915046 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.914970    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10261a6e-338b-4c67-99bd-6921a24e051b-tmp-dir\") pod \"dns-default-rn9mr\" (UID: \"10261a6e-338b-4c67-99bd-6921a24e051b\") " pod="openshift-dns/dns-default-rn9mr"
Sep 11 21:53:01.917748 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.917724    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 11 21:53:01.917864 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.917728    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 11 21:53:01.917980 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.917967    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 11 21:53:01.918082 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.918069    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-x6t9c\""
Sep 11 21:53:01.918363 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.918322    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 11 21:53:01.918627 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.918611    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3cdabfb8-9e10-4fe2-9ba4-4301f2ea9fd0-cert\") pod \"ingress-canary-dnlj8\" (UID: \"3cdabfb8-9e10-4fe2-9ba4-4301f2ea9fd0\") " pod="openshift-ingress-canary/ingress-canary-dnlj8"
Sep 11 21:53:01.923397 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.923373    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2m4h\" (UniqueName: \"kubernetes.io/projected/3cdabfb8-9e10-4fe2-9ba4-4301f2ea9fd0-kube-api-access-q2m4h\") pod \"ingress-canary-dnlj8\" (UID: \"3cdabfb8-9e10-4fe2-9ba4-4301f2ea9fd0\") " pod="openshift-ingress-canary/ingress-canary-dnlj8"
Sep 11 21:53:01.923990 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:01.923972    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlrk6\" (UniqueName: \"kubernetes.io/projected/10261a6e-338b-4c67-99bd-6921a24e051b-kube-api-access-hlrk6\") pod \"dns-default-rn9mr\" (UID: \"10261a6e-338b-4c67-99bd-6921a24e051b\") " pod="openshift-dns/dns-default-rn9mr"
Sep 11 21:53:02.012279 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.012196    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dnlj8"
Sep 11 21:53:02.015480 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.015454    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9qjm\" (UID: \"3fdc3282-48a9-49ae-a5a4-5d60fb93639a\") " pod="openshift-insights/insights-runtime-extractor-c9qjm"
Sep 11 21:53:02.015564 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.015515    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c9qjm\" (UID: \"3fdc3282-48a9-49ae-a5a4-5d60fb93639a\") " pod="openshift-insights/insights-runtime-extractor-c9qjm"
Sep 11 21:53:02.015564 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.015530    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7g7l\" (UniqueName: \"kubernetes.io/projected/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-kube-api-access-l7g7l\") pod \"insights-runtime-extractor-c9qjm\" (UID: \"3fdc3282-48a9-49ae-a5a4-5d60fb93639a\") " pod="openshift-insights/insights-runtime-extractor-c9qjm"
Sep 11 21:53:02.015564 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.015549    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e199d531-f344-4c9f-90b0-dad821bff99b-signing-cabundle\") pod \"service-ca-568869fdc5-6xt8x\" (UID: \"e199d531-f344-4c9f-90b0-dad821bff99b\") " pod="openshift-service-ca/service-ca-568869fdc5-6xt8x"
Sep 11 21:53:02.015713 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:02.015593    2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:53:02.015713 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:02.015630    2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:53:02.015713 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:02.015694    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-kube-rbac-proxy-cm podName:3fdc3282-48a9-49ae-a5a4-5d60fb93639a nodeName:}" failed. No retries permitted until 2026-09-11 21:53:02.51567832 +0000 UTC m=+35.690754671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c9qjm" (UID: "3fdc3282-48a9-49ae-a5a4-5d60fb93639a") : configmap "kube-rbac-proxy" not found
Sep 11 21:53:02.015824 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:02.015759    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-insights-runtime-extractor-tls podName:3fdc3282-48a9-49ae-a5a4-5d60fb93639a nodeName:}" failed. No retries permitted until 2026-09-11 21:53:02.515742538 +0000 UTC m=+35.690818891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c9qjm" (UID: "3fdc3282-48a9-49ae-a5a4-5d60fb93639a") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:53:02.015824 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.015795    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-data-volume\") pod \"insights-runtime-extractor-c9qjm\" (UID: \"3fdc3282-48a9-49ae-a5a4-5d60fb93639a\") " pod="openshift-insights/insights-runtime-extractor-c9qjm"
Sep 11 21:53:02.015824 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.015817    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-crio-socket\") pod \"insights-runtime-extractor-c9qjm\" (UID: \"3fdc3282-48a9-49ae-a5a4-5d60fb93639a\") " pod="openshift-insights/insights-runtime-extractor-c9qjm"
Sep 11 21:53:02.015927 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.015834    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e199d531-f344-4c9f-90b0-dad821bff99b-signing-key\") pod \"service-ca-568869fdc5-6xt8x\" (UID: \"e199d531-f344-4c9f-90b0-dad821bff99b\") " pod="openshift-service-ca/service-ca-568869fdc5-6xt8x"
Sep 11 21:53:02.015927 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.015851    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crzc2\" (UniqueName: \"kubernetes.io/projected/e199d531-f344-4c9f-90b0-dad821bff99b-kube-api-access-crzc2\") pod \"service-ca-568869fdc5-6xt8x\" (UID: \"e199d531-f344-4c9f-90b0-dad821bff99b\") " pod="openshift-service-ca/service-ca-568869fdc5-6xt8x"
Sep 11 21:53:02.016068 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.016048    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-crio-socket\") pod \"insights-runtime-extractor-c9qjm\" (UID: \"3fdc3282-48a9-49ae-a5a4-5d60fb93639a\") " pod="openshift-insights/insights-runtime-extractor-c9qjm"
Sep 11 21:53:02.016123 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.016109    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-data-volume\") pod \"insights-runtime-extractor-c9qjm\" (UID: \"3fdc3282-48a9-49ae-a5a4-5d60fb93639a\") " pod="openshift-insights/insights-runtime-extractor-c9qjm"
Sep 11 21:53:02.024977 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.024954    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7g7l\" (UniqueName: \"kubernetes.io/projected/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-kube-api-access-l7g7l\") pod \"insights-runtime-extractor-c9qjm\" (UID: \"3fdc3282-48a9-49ae-a5a4-5d60fb93639a\") " pod="openshift-insights/insights-runtime-extractor-c9qjm"
Sep 11 21:53:02.117122 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.116920    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e199d531-f344-4c9f-90b0-dad821bff99b-signing-cabundle\") pod \"service-ca-568869fdc5-6xt8x\" (UID: \"e199d531-f344-4c9f-90b0-dad821bff99b\") " pod="openshift-service-ca/service-ca-568869fdc5-6xt8x"
Sep 11 21:53:02.117241 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.117185    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e199d531-f344-4c9f-90b0-dad821bff99b-signing-key\") pod \"service-ca-568869fdc5-6xt8x\" (UID: \"e199d531-f344-4c9f-90b0-dad821bff99b\") " pod="openshift-service-ca/service-ca-568869fdc5-6xt8x"
Sep 11 21:53:02.117241 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.117222    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crzc2\" (UniqueName: \"kubernetes.io/projected/e199d531-f344-4c9f-90b0-dad821bff99b-kube-api-access-crzc2\") pod \"service-ca-568869fdc5-6xt8x\" (UID: \"e199d531-f344-4c9f-90b0-dad821bff99b\") " pod="openshift-service-ca/service-ca-568869fdc5-6xt8x"
Sep 11 21:53:02.118014 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.117990    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e199d531-f344-4c9f-90b0-dad821bff99b-signing-cabundle\") pod \"service-ca-568869fdc5-6xt8x\" (UID: \"e199d531-f344-4c9f-90b0-dad821bff99b\") " pod="openshift-service-ca/service-ca-568869fdc5-6xt8x"
Sep 11 21:53:02.120656 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.120631    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e199d531-f344-4c9f-90b0-dad821bff99b-signing-key\") pod \"service-ca-568869fdc5-6xt8x\" (UID: \"e199d531-f344-4c9f-90b0-dad821bff99b\") " pod="openshift-service-ca/service-ca-568869fdc5-6xt8x"
Sep 11 21:53:02.125638 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.125619    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crzc2\" (UniqueName: \"kubernetes.io/projected/e199d531-f344-4c9f-90b0-dad821bff99b-kube-api-access-crzc2\") pod \"service-ca-568869fdc5-6xt8x\" (UID: \"e199d531-f344-4c9f-90b0-dad821bff99b\") " pod="openshift-service-ca/service-ca-568869fdc5-6xt8x"
Sep 11 21:53:02.232732 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.232701    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-6xt8x"
Sep 11 21:53:02.328943 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.328913    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dnlj8"]
Sep 11 21:53:02.330214 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.330189    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:53:02.330314 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.330295    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:53:02.330392 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.330347    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:53:02.332822 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:53:02.332798    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cdabfb8_9e10_4fe2_9ba4_4301f2ea9fd0.slice/crio-7bc4efe07106c0a32d9746f177675a97a21d275f5c2041d961421c9da26bb1b7 WatchSource:0}: Error finding container 7bc4efe07106c0a32d9746f177675a97a21d275f5c2041d961421c9da26bb1b7: Status 404 returned error can't find the container with id 7bc4efe07106c0a32d9746f177675a97a21d275f5c2041d961421c9da26bb1b7
Sep 11 21:53:02.333297 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.333276    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 21:53:02.333297 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.333288    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 21:53:02.333451 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.333288    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 21:53:02.333451 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.333393    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4bfmc\""
Sep 11 21:53:02.334539 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.334522    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qsk4f\""
Sep 11 21:53:02.334618 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.334562    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 21:53:02.419348 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.419299    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10261a6e-338b-4c67-99bd-6921a24e051b-metrics-tls\") pod \"dns-default-rn9mr\" (UID: \"10261a6e-338b-4c67-99bd-6921a24e051b\") " pod="openshift-dns/dns-default-rn9mr"
Sep 11 21:53:02.419548 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:02.419455    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:53:02.419548 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:02.419523    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10261a6e-338b-4c67-99bd-6921a24e051b-metrics-tls podName:10261a6e-338b-4c67-99bd-6921a24e051b nodeName:}" failed. No retries permitted until 2026-09-11 21:53:03.419504198 +0000 UTC m=+36.594580554 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10261a6e-338b-4c67-99bd-6921a24e051b-metrics-tls") pod "dns-default-rn9mr" (UID: "10261a6e-338b-4c67-99bd-6921a24e051b") : secret "dns-default-metrics-tls" not found
Sep 11 21:53:02.516994 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.516919    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dnlj8" event={"ID":"3cdabfb8-9e10-4fe2-9ba4-4301f2ea9fd0","Type":"ContainerStarted","Data":"7bc4efe07106c0a32d9746f177675a97a21d275f5c2041d961421c9da26bb1b7"}
Sep 11 21:53:02.519656 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.519627    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9qjm\" (UID: \"3fdc3282-48a9-49ae-a5a4-5d60fb93639a\") " pod="openshift-insights/insights-runtime-extractor-c9qjm"
Sep 11 21:53:02.519767 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.519688    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c9qjm\" (UID: \"3fdc3282-48a9-49ae-a5a4-5d60fb93639a\") " pod="openshift-insights/insights-runtime-extractor-c9qjm"
Sep 11 21:53:02.519854 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:02.519765    2587 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 21:53:02.519854 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:02.519770    2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:53:02.519854 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:02.519820    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-kube-rbac-proxy-cm podName:3fdc3282-48a9-49ae-a5a4-5d60fb93639a nodeName:}" failed. No retries permitted until 2026-09-11 21:53:03.519800983 +0000 UTC m=+36.694877333 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c9qjm" (UID: "3fdc3282-48a9-49ae-a5a4-5d60fb93639a") : configmap "kube-rbac-proxy" not found
Sep 11 21:53:02.519854 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:02.519834    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-insights-runtime-extractor-tls podName:3fdc3282-48a9-49ae-a5a4-5d60fb93639a nodeName:}" failed. No retries permitted until 2026-09-11 21:53:03.519827596 +0000 UTC m=+36.694903944 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c9qjm" (UID: "3fdc3282-48a9-49ae-a5a4-5d60fb93639a") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:53:02.520482 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.520465    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lzrhd" event={"ID":"5d5e4814-3128-4082-aca2-5bbfa715b2e4","Type":"ContainerStarted","Data":"4c2628d98abdc7bb96fcab44b1ad690c9982c09a77918d0098e08385bedd9636"}
Sep 11 21:53:02.532307 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.532286    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-6xt8x"]
Sep 11 21:53:02.535341 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:53:02.535312    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode199d531_f344_4c9f_90b0_dad821bff99b.slice/crio-c319cb88178f036f74abeb8337e9c71059b7440207b19dfbed4cefb796ae1f94 WatchSource:0}: Error finding container c319cb88178f036f74abeb8337e9c71059b7440207b19dfbed4cefb796ae1f94: Status 404 returned error can't find the container with id c319cb88178f036f74abeb8337e9c71059b7440207b19dfbed4cefb796ae1f94
Sep 11 21:53:02.546794 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:02.546752    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lzrhd" podStartSLOduration=7.9575295310000005 podStartE2EDuration="35.54673799s" totalImagesPullingTime="27.589208459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:02.546362187 +0000 UTC m=+35.721438574" watchObservedRunningTime="2026-09-11 21:53:02.54673799 +0000 UTC m=+35.721814360"
Sep 11 21:53:03.428928 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:03.428886    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10261a6e-338b-4c67-99bd-6921a24e051b-metrics-tls\") pod \"dns-default-rn9mr\" (UID: \"10261a6e-338b-4c67-99bd-6921a24e051b\") " pod="openshift-dns/dns-default-rn9mr"
Sep 11 21:53:03.429732 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:03.429051    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:53:03.429732 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:03.429129    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10261a6e-338b-4c67-99bd-6921a24e051b-metrics-tls podName:10261a6e-338b-4c67-99bd-6921a24e051b nodeName:}" failed. No retries permitted until 2026-09-11 21:53:05.429111919 +0000 UTC m=+38.604188274 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10261a6e-338b-4c67-99bd-6921a24e051b-metrics-tls") pod "dns-default-rn9mr" (UID: "10261a6e-338b-4c67-99bd-6921a24e051b") : secret "dns-default-metrics-tls" not found
Sep 11 21:53:03.525501 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:03.525357    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-6xt8x" event={"ID":"e199d531-f344-4c9f-90b0-dad821bff99b","Type":"ContainerStarted","Data":"c319cb88178f036f74abeb8337e9c71059b7440207b19dfbed4cefb796ae1f94"}
Sep 11 21:53:03.531420 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:03.530155    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9qjm\" (UID: \"3fdc3282-48a9-49ae-a5a4-5d60fb93639a\") " pod="openshift-insights/insights-runtime-extractor-c9qjm"
Sep 11 21:53:03.531420 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:03.530239    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c9qjm\" (UID: \"3fdc3282-48a9-49ae-a5a4-5d60fb93639a\") " pod="openshift-insights/insights-runtime-extractor-c9qjm"
Sep 11 21:53:03.531420 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:03.530907    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c9qjm\" (UID: \"3fdc3282-48a9-49ae-a5a4-5d60fb93639a\") " pod="openshift-insights/insights-runtime-extractor-c9qjm"
Sep 11 21:53:03.531420 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:03.531011    2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:53:03.531420 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:03.531069    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-insights-runtime-extractor-tls podName:3fdc3282-48a9-49ae-a5a4-5d60fb93639a nodeName:}" failed. No retries permitted until 2026-09-11 21:53:05.531051805 +0000 UTC m=+38.706128159 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c9qjm" (UID: "3fdc3282-48a9-49ae-a5a4-5d60fb93639a") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:53:05.447615 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:05.447577    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10261a6e-338b-4c67-99bd-6921a24e051b-metrics-tls\") pod \"dns-default-rn9mr\" (UID: \"10261a6e-338b-4c67-99bd-6921a24e051b\") " pod="openshift-dns/dns-default-rn9mr"
Sep 11 21:53:05.448068 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:05.447699    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:53:05.448068 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:05.447752    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10261a6e-338b-4c67-99bd-6921a24e051b-metrics-tls podName:10261a6e-338b-4c67-99bd-6921a24e051b nodeName:}" failed. No retries permitted until 2026-09-11 21:53:09.447738321 +0000 UTC m=+42.622814668 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10261a6e-338b-4c67-99bd-6921a24e051b-metrics-tls") pod "dns-default-rn9mr" (UID: "10261a6e-338b-4c67-99bd-6921a24e051b") : secret "dns-default-metrics-tls" not found
Sep 11 21:53:05.548484 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:05.548456    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9qjm\" (UID: \"3fdc3282-48a9-49ae-a5a4-5d60fb93639a\") " pod="openshift-insights/insights-runtime-extractor-c9qjm"
Sep 11 21:53:05.548630 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:05.548611    2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:53:05.548687 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:05.548673    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-insights-runtime-extractor-tls podName:3fdc3282-48a9-49ae-a5a4-5d60fb93639a nodeName:}" failed. No retries permitted until 2026-09-11 21:53:09.54865811 +0000 UTC m=+42.723734457 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c9qjm" (UID: "3fdc3282-48a9-49ae-a5a4-5d60fb93639a") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:53:06.534513 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:06.534234    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dnlj8" event={"ID":"3cdabfb8-9e10-4fe2-9ba4-4301f2ea9fd0","Type":"ContainerStarted","Data":"60f847f1b49b8a34f7b9c189964ac7cf8d23f2f3bc4599abd9d4a27a485699e9"}
Sep 11 21:53:06.535855 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:06.535834    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-6xt8x" event={"ID":"e199d531-f344-4c9f-90b0-dad821bff99b","Type":"ContainerStarted","Data":"3a4817dc63b6f42eeabe34dbc46a22c6c5a3679fab36124cb34d99f239dc86e0"}
Sep 11 21:53:06.552396 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:06.552344    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dnlj8" podStartSLOduration=2.37420875 podStartE2EDuration="5.552307085s" totalImagesPullingTime="3.178098335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:53:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:06.552093993 +0000 UTC m=+39.727170383" watchObservedRunningTime="2026-09-11 21:53:06.552307085 +0000 UTC m=+39.727383457"
Sep 11 21:53:06.568015 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:06.567959    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-6xt8x" podStartSLOduration=2.586481894 podStartE2EDuration="5.567940288s" totalImagesPullingTime="2.981458394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:53:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:06.567676754 +0000 UTC m=+39.742753153" watchObservedRunningTime="2026-09-11 21:53:06.567940288 +0000 UTC m=+39.743016658"
Sep 11 21:53:09.478741 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:09.478696    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10261a6e-338b-4c67-99bd-6921a24e051b-metrics-tls\") pod \"dns-default-rn9mr\" (UID: \"10261a6e-338b-4c67-99bd-6921a24e051b\") " pod="openshift-dns/dns-default-rn9mr"
Sep 11 21:53:09.479129 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:09.478857    2587 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 21:53:09.479129 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:09.478930    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10261a6e-338b-4c67-99bd-6921a24e051b-metrics-tls podName:10261a6e-338b-4c67-99bd-6921a24e051b nodeName:}" failed. No retries permitted until 2026-09-11 21:53:17.478912961 +0000 UTC m=+50.653989309 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/10261a6e-338b-4c67-99bd-6921a24e051b-metrics-tls") pod "dns-default-rn9mr" (UID: "10261a6e-338b-4c67-99bd-6921a24e051b") : secret "dns-default-metrics-tls" not found
Sep 11 21:53:09.579257 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:09.579216    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9qjm\" (UID: \"3fdc3282-48a9-49ae-a5a4-5d60fb93639a\") " pod="openshift-insights/insights-runtime-extractor-c9qjm"
Sep 11 21:53:09.579483 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:09.579409    2587 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 21:53:09.579555 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:09.579494    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-insights-runtime-extractor-tls podName:3fdc3282-48a9-49ae-a5a4-5d60fb93639a nodeName:}" failed. No retries permitted until 2026-09-11 21:53:17.57947173 +0000 UTC m=+50.754548080 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c9qjm" (UID: "3fdc3282-48a9-49ae-a5a4-5d60fb93639a") : secret "insights-runtime-extractor-tls" not found
Sep 11 21:53:17.540903 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:17.540860    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10261a6e-338b-4c67-99bd-6921a24e051b-metrics-tls\") pod \"dns-default-rn9mr\" (UID: \"10261a6e-338b-4c67-99bd-6921a24e051b\") " pod="openshift-dns/dns-default-rn9mr"
Sep 11 21:53:17.543698 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:17.543675    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/10261a6e-338b-4c67-99bd-6921a24e051b-metrics-tls\") pod \"dns-default-rn9mr\" (UID: \"10261a6e-338b-4c67-99bd-6921a24e051b\") " pod="openshift-dns/dns-default-rn9mr"
Sep 11 21:53:17.636660 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:17.636632    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rn9mr"
Sep 11 21:53:17.641574 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:17.641549    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9qjm\" (UID: \"3fdc3282-48a9-49ae-a5a4-5d60fb93639a\") " pod="openshift-insights/insights-runtime-extractor-c9qjm"
Sep 11 21:53:17.644305 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:17.644285    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3fdc3282-48a9-49ae-a5a4-5d60fb93639a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c9qjm\" (UID: \"3fdc3282-48a9-49ae-a5a4-5d60fb93639a\") " pod="openshift-insights/insights-runtime-extractor-c9qjm"
Sep 11 21:53:17.653864 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:17.653835    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c9qjm"
Sep 11 21:53:17.907869 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:17.907795    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rn9mr"]
Sep 11 21:53:17.911809 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:53:17.911773    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10261a6e_338b_4c67_99bd_6921a24e051b.slice/crio-ed490ebe88029f6fb9cf3f5e5ce3a76e358b2f66941be0b5e8ad1ba243016e75 WatchSource:0}: Error finding container ed490ebe88029f6fb9cf3f5e5ce3a76e358b2f66941be0b5e8ad1ba243016e75: Status 404 returned error can't find the container with id ed490ebe88029f6fb9cf3f5e5ce3a76e358b2f66941be0b5e8ad1ba243016e75
Sep 11 21:53:17.927404 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:17.927376    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c9qjm"]
Sep 11 21:53:17.930262 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:53:17.930236    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fdc3282_48a9_49ae_a5a4_5d60fb93639a.slice/crio-20bcf02e8f8a92a8c34ac5ecc40806c5b1eb22d9a4941b4c29446de0058a6598 WatchSource:0}: Error finding container 20bcf02e8f8a92a8c34ac5ecc40806c5b1eb22d9a4941b4c29446de0058a6598: Status 404 returned error can't find the container with id 20bcf02e8f8a92a8c34ac5ecc40806c5b1eb22d9a4941b4c29446de0058a6598
Sep 11 21:53:18.569005 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:18.568965    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rn9mr" event={"ID":"10261a6e-338b-4c67-99bd-6921a24e051b","Type":"ContainerStarted","Data":"ed490ebe88029f6fb9cf3f5e5ce3a76e358b2f66941be0b5e8ad1ba243016e75"}
Sep 11 21:53:18.570660 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:18.570633    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c9qjm" event={"ID":"3fdc3282-48a9-49ae-a5a4-5d60fb93639a","Type":"ContainerStarted","Data":"31a2486b96b8b02f83b774fec763dc4d714a42d4f2402ed99e199088cef5bb9f"}
Sep 11 21:53:18.570761 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:18.570680    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c9qjm" event={"ID":"3fdc3282-48a9-49ae-a5a4-5d60fb93639a","Type":"ContainerStarted","Data":"20bcf02e8f8a92a8c34ac5ecc40806c5b1eb22d9a4941b4c29446de0058a6598"}
Sep 11 21:53:19.574730 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:19.574698    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c9qjm" event={"ID":"3fdc3282-48a9-49ae-a5a4-5d60fb93639a","Type":"ContainerStarted","Data":"a9b0899d07eb7aef0e60c0e488f80ccfa81f00e96fa06971ea9eac950cccd12d"}
Sep 11 21:53:20.585381 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:20.585336    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rn9mr" event={"ID":"10261a6e-338b-4c67-99bd-6921a24e051b","Type":"ContainerStarted","Data":"6d4772c2096a8d9d4f3960c986ad92ed8d4022cfa2444236a160b05bfab94e91"}
Sep 11 21:53:20.585813 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:20.585429    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rn9mr"
Sep 11 21:53:20.585813 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:20.585457    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rn9mr" event={"ID":"10261a6e-338b-4c67-99bd-6921a24e051b","Type":"ContainerStarted","Data":"fbda7c910cd3986f1fbc566624eda67a024cba115015b95073da24c77505ba67"}
Sep 11 21:53:20.587465 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:20.587440    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c9qjm" event={"ID":"3fdc3282-48a9-49ae-a5a4-5d60fb93639a","Type":"ContainerStarted","Data":"0e63de2b1a808f06ed3fc4fda48576038bb908ece12de16dbbc674e60bb2829b"}
Sep 11 21:53:20.603208 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:20.603163    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rn9mr" podStartSLOduration=17.527744703 podStartE2EDuration="19.603150908s" totalImagesPullingTime="2.075406205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:53:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:20.601251705 +0000 UTC m=+53.776328086" watchObservedRunningTime="2026-09-11 21:53:20.603150908 +0000 UTC m=+53.778227268"
Sep 11 21:53:20.617454 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:20.617412    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-c9qjm" podStartSLOduration=17.685992682 podStartE2EDuration="19.617398608s" totalImagesPullingTime="1.931405926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:53:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:20.616822974 +0000 UTC m=+53.791899375" watchObservedRunningTime="2026-09-11 21:53:20.617398608 +0000 UTC m=+53.792474977"
Sep 11 21:53:21.288970 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.288937    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-wplsx"]
Sep 11 21:53:21.291860 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.291841    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84c8f5cd7b-hd4bb"]
Sep 11 21:53:21.291986 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.291969    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wplsx"
Sep 11 21:53:21.294564 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.294543    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bb5b4db8-vzdsr"]
Sep 11 21:53:21.294676 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.294615    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84c8f5cd7b-hd4bb"
Sep 11 21:53:21.296274 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.296253    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hxnsm\""
Sep 11 21:53:21.296518 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.296503    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 11 21:53:21.296601 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.296540    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 11 21:53:21.297235 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.297214    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bb5b4db8-vzdsr"
Sep 11 21:53:21.297614 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.297597    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-m79gz\""
Sep 11 21:53:21.298784 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.298715    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 21:53:21.298864 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.298817    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 11 21:53:21.298985 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.298962    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 21:53:21.300118 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.300100    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 21:53:21.300216 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.300165    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 11 21:53:21.314820 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.314800    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-wplsx"]
Sep 11 21:53:21.318761 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.318731    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84c8f5cd7b-hd4bb"]
Sep 11 21:53:21.319705 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.319685    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bb5b4db8-vzdsr"]
Sep 11 21:53:21.366837 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.366811    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d10cc55c-a4d2-4c30-84d9-15ced74f00b3-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-wplsx\" (UID: \"d10cc55c-a4d2-4c30-84d9-15ced74f00b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wplsx"
Sep 11 21:53:21.366837 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.366841    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f5dbbfd-df55-47ec-b210-0dc2b22142e9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-84c8f5cd7b-hd4bb\" (UID: \"6f5dbbfd-df55-47ec-b210-0dc2b22142e9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84c8f5cd7b-hd4bb"
Sep 11 21:53:21.367038 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.366859    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbgqp\" (UniqueName: \"kubernetes.io/projected/6f5dbbfd-df55-47ec-b210-0dc2b22142e9-kube-api-access-hbgqp\") pod \"managed-serviceaccount-addon-agent-84c8f5cd7b-hd4bb\" (UID: \"6f5dbbfd-df55-47ec-b210-0dc2b22142e9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84c8f5cd7b-hd4bb"
Sep 11 21:53:21.367038 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.366874    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/08c2350e-f194-4848-b674-08918601b698-klusterlet-config\") pod \"klusterlet-addon-workmgr-59bb5b4db8-vzdsr\" (UID: \"08c2350e-f194-4848-b674-08918601b698\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bb5b4db8-vzdsr"
Sep 11 21:53:21.367038 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.366900    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d10cc55c-a4d2-4c30-84d9-15ced74f00b3-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-wplsx\" (UID: \"d10cc55c-a4d2-4c30-84d9-15ced74f00b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wplsx"
Sep 11 21:53:21.367038 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.366920    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08c2350e-f194-4848-b674-08918601b698-tmp\") pod \"klusterlet-addon-workmgr-59bb5b4db8-vzdsr\" (UID: \"08c2350e-f194-4848-b674-08918601b698\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bb5b4db8-vzdsr"
Sep 11 21:53:21.367038 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.366962    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6s92\" (UniqueName: \"kubernetes.io/projected/08c2350e-f194-4848-b674-08918601b698-kube-api-access-z6s92\") pod \"klusterlet-addon-workmgr-59bb5b4db8-vzdsr\" (UID: \"08c2350e-f194-4848-b674-08918601b698\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bb5b4db8-vzdsr"
Sep 11 21:53:21.392540 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.392507    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-6p4cp"]
Sep 11 21:53:21.395464 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.395448    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-6p4cp"
Sep 11 21:53:21.402056 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.402034    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-g9768\""
Sep 11 21:53:21.402354 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.402322    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 21:53:21.403464 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.403447    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 21:53:21.409601 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.409577    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-6p4cp"]
Sep 11 21:53:21.467909 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.467870    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d10cc55c-a4d2-4c30-84d9-15ced74f00b3-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-wplsx\" (UID: \"d10cc55c-a4d2-4c30-84d9-15ced74f00b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wplsx"
Sep 11 21:53:21.468091 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.467915    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f5dbbfd-df55-47ec-b210-0dc2b22142e9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-84c8f5cd7b-hd4bb\" (UID: \"6f5dbbfd-df55-47ec-b210-0dc2b22142e9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84c8f5cd7b-hd4bb"
Sep 11 21:53:21.468091 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.467945    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbgqp\" (UniqueName: \"kubernetes.io/projected/6f5dbbfd-df55-47ec-b210-0dc2b22142e9-kube-api-access-hbgqp\") pod \"managed-serviceaccount-addon-agent-84c8f5cd7b-hd4bb\" (UID: \"6f5dbbfd-df55-47ec-b210-0dc2b22142e9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84c8f5cd7b-hd4bb"
Sep 11 21:53:21.468091 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.467968    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/08c2350e-f194-4848-b674-08918601b698-klusterlet-config\") pod \"klusterlet-addon-workmgr-59bb5b4db8-vzdsr\" (UID: \"08c2350e-f194-4848-b674-08918601b698\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bb5b4db8-vzdsr"
Sep 11 21:53:21.468091 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.468011    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d10cc55c-a4d2-4c30-84d9-15ced74f00b3-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-wplsx\" (UID: \"d10cc55c-a4d2-4c30-84d9-15ced74f00b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wplsx"
Sep 11 21:53:21.468311 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.468132    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08c2350e-f194-4848-b674-08918601b698-tmp\") pod \"klusterlet-addon-workmgr-59bb5b4db8-vzdsr\" (UID: \"08c2350e-f194-4848-b674-08918601b698\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bb5b4db8-vzdsr"
Sep 11 21:53:21.468311 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.468186    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6s92\" (UniqueName: \"kubernetes.io/projected/08c2350e-f194-4848-b674-08918601b698-kube-api-access-z6s92\") pod \"klusterlet-addon-workmgr-59bb5b4db8-vzdsr\" (UID: \"08c2350e-f194-4848-b674-08918601b698\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bb5b4db8-vzdsr"
Sep 11 21:53:21.468311 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.468224    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5wgf\" (UniqueName: \"kubernetes.io/projected/8c10e945-fa90-4b6a-b5a8-423c05d6db4d-kube-api-access-s5wgf\") pod \"downloads-76f99584f4-6p4cp\" (UID: \"8c10e945-fa90-4b6a-b5a8-423c05d6db4d\") " pod="openshift-console/downloads-76f99584f4-6p4cp"
Sep 11 21:53:21.468604 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.468571    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08c2350e-f194-4848-b674-08918601b698-tmp\") pod \"klusterlet-addon-workmgr-59bb5b4db8-vzdsr\" (UID: \"08c2350e-f194-4848-b674-08918601b698\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bb5b4db8-vzdsr"
Sep 11 21:53:21.468837 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.468810    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d10cc55c-a4d2-4c30-84d9-15ced74f00b3-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-wplsx\" (UID: \"d10cc55c-a4d2-4c30-84d9-15ced74f00b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wplsx"
Sep 11 21:53:21.470517 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.470491    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d10cc55c-a4d2-4c30-84d9-15ced74f00b3-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-wplsx\" (UID: \"d10cc55c-a4d2-4c30-84d9-15ced74f00b3\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wplsx"
Sep 11 21:53:21.470597 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.470515    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/08c2350e-f194-4848-b674-08918601b698-klusterlet-config\") pod \"klusterlet-addon-workmgr-59bb5b4db8-vzdsr\" (UID: \"08c2350e-f194-4848-b674-08918601b698\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bb5b4db8-vzdsr"
Sep 11 21:53:21.470597 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.470532    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f5dbbfd-df55-47ec-b210-0dc2b22142e9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-84c8f5cd7b-hd4bb\" (UID: \"6f5dbbfd-df55-47ec-b210-0dc2b22142e9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84c8f5cd7b-hd4bb"
Sep 11 21:53:21.478914 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.478891    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6s92\" (UniqueName: \"kubernetes.io/projected/08c2350e-f194-4848-b674-08918601b698-kube-api-access-z6s92\") pod \"klusterlet-addon-workmgr-59bb5b4db8-vzdsr\" (UID: \"08c2350e-f194-4848-b674-08918601b698\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bb5b4db8-vzdsr"
Sep 11 21:53:21.479003 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.478920    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbgqp\" (UniqueName: \"kubernetes.io/projected/6f5dbbfd-df55-47ec-b210-0dc2b22142e9-kube-api-access-hbgqp\") pod \"managed-serviceaccount-addon-agent-84c8f5cd7b-hd4bb\" (UID: \"6f5dbbfd-df55-47ec-b210-0dc2b22142e9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84c8f5cd7b-hd4bb"
Sep 11 21:53:21.493718 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.493695    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56b94bddb8-spm56"]
Sep 11 21:53:21.501185 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.501163    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.503988 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.503957    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 21:53:21.504091 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.503989    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 21:53:21.504091 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.503989    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 21:53:21.504225 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.504185    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-m5hpf\""
Sep 11 21:53:21.509920 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.509900    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56b94bddb8-spm56"]
Sep 11 21:53:21.510114 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.510099    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 21:53:21.568621 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.568538    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5wgf\" (UniqueName: \"kubernetes.io/projected/8c10e945-fa90-4b6a-b5a8-423c05d6db4d-kube-api-access-s5wgf\") pod \"downloads-76f99584f4-6p4cp\" (UID: \"8c10e945-fa90-4b6a-b5a8-423c05d6db4d\") " pod="openshift-console/downloads-76f99584f4-6p4cp"
Sep 11 21:53:21.568621 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.568574    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/159a6253-516b-4444-81d9-d7daa7b247af-registry-certificates\") pod \"image-registry-56b94bddb8-spm56\" (UID: \"159a6253-516b-4444-81d9-d7daa7b247af\") " pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.568621 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.568592    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/159a6253-516b-4444-81d9-d7daa7b247af-ca-trust-extracted\") pod \"image-registry-56b94bddb8-spm56\" (UID: \"159a6253-516b-4444-81d9-d7daa7b247af\") " pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.568880 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.568694    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/159a6253-516b-4444-81d9-d7daa7b247af-bound-sa-token\") pod \"image-registry-56b94bddb8-spm56\" (UID: \"159a6253-516b-4444-81d9-d7daa7b247af\") " pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.568880 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.568732    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/159a6253-516b-4444-81d9-d7daa7b247af-image-registry-private-configuration\") pod \"image-registry-56b94bddb8-spm56\" (UID: \"159a6253-516b-4444-81d9-d7daa7b247af\") " pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.568880 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.568794    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/159a6253-516b-4444-81d9-d7daa7b247af-trusted-ca\") pod \"image-registry-56b94bddb8-spm56\" (UID: \"159a6253-516b-4444-81d9-d7daa7b247af\") " pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.568880 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.568832    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj75s\" (UniqueName: \"kubernetes.io/projected/159a6253-516b-4444-81d9-d7daa7b247af-kube-api-access-bj75s\") pod \"image-registry-56b94bddb8-spm56\" (UID: \"159a6253-516b-4444-81d9-d7daa7b247af\") " pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.568880 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.568861    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/159a6253-516b-4444-81d9-d7daa7b247af-registry-tls\") pod \"image-registry-56b94bddb8-spm56\" (UID: \"159a6253-516b-4444-81d9-d7daa7b247af\") " pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.568880 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.568876    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/159a6253-516b-4444-81d9-d7daa7b247af-installation-pull-secrets\") pod \"image-registry-56b94bddb8-spm56\" (UID: \"159a6253-516b-4444-81d9-d7daa7b247af\") " pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.577877 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.577850    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5wgf\" (UniqueName: \"kubernetes.io/projected/8c10e945-fa90-4b6a-b5a8-423c05d6db4d-kube-api-access-s5wgf\") pod \"downloads-76f99584f4-6p4cp\" (UID: \"8c10e945-fa90-4b6a-b5a8-423c05d6db4d\") " pod="openshift-console/downloads-76f99584f4-6p4cp"
Sep 11 21:53:21.602271 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.602247    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wplsx"
Sep 11 21:53:21.625222 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.625199    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84c8f5cd7b-hd4bb"
Sep 11 21:53:21.630719 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.630687    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bb5b4db8-vzdsr"
Sep 11 21:53:21.670367 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.670233    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/159a6253-516b-4444-81d9-d7daa7b247af-registry-tls\") pod \"image-registry-56b94bddb8-spm56\" (UID: \"159a6253-516b-4444-81d9-d7daa7b247af\") " pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.670367 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.670279    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/159a6253-516b-4444-81d9-d7daa7b247af-installation-pull-secrets\") pod \"image-registry-56b94bddb8-spm56\" (UID: \"159a6253-516b-4444-81d9-d7daa7b247af\") " pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.672122 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.670745    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/159a6253-516b-4444-81d9-d7daa7b247af-registry-certificates\") pod \"image-registry-56b94bddb8-spm56\" (UID: \"159a6253-516b-4444-81d9-d7daa7b247af\") " pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.672122 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.670789    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/159a6253-516b-4444-81d9-d7daa7b247af-ca-trust-extracted\") pod \"image-registry-56b94bddb8-spm56\" (UID: \"159a6253-516b-4444-81d9-d7daa7b247af\") " pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.672122 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.670828    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/159a6253-516b-4444-81d9-d7daa7b247af-bound-sa-token\") pod \"image-registry-56b94bddb8-spm56\" (UID: \"159a6253-516b-4444-81d9-d7daa7b247af\") " pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.672122 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.670925    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/159a6253-516b-4444-81d9-d7daa7b247af-image-registry-private-configuration\") pod \"image-registry-56b94bddb8-spm56\" (UID: \"159a6253-516b-4444-81d9-d7daa7b247af\") " pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.672122 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.671005    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/159a6253-516b-4444-81d9-d7daa7b247af-trusted-ca\") pod \"image-registry-56b94bddb8-spm56\" (UID: \"159a6253-516b-4444-81d9-d7daa7b247af\") " pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.672122 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.671033    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj75s\" (UniqueName: \"kubernetes.io/projected/159a6253-516b-4444-81d9-d7daa7b247af-kube-api-access-bj75s\") pod \"image-registry-56b94bddb8-spm56\" (UID: \"159a6253-516b-4444-81d9-d7daa7b247af\") " pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.672122 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.671848    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/159a6253-516b-4444-81d9-d7daa7b247af-ca-trust-extracted\") pod \"image-registry-56b94bddb8-spm56\" (UID: \"159a6253-516b-4444-81d9-d7daa7b247af\") " pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.672555 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.672408    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/159a6253-516b-4444-81d9-d7daa7b247af-trusted-ca\") pod \"image-registry-56b94bddb8-spm56\" (UID: \"159a6253-516b-4444-81d9-d7daa7b247af\") " pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.672614 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.672581    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/159a6253-516b-4444-81d9-d7daa7b247af-registry-certificates\") pod \"image-registry-56b94bddb8-spm56\" (UID: \"159a6253-516b-4444-81d9-d7daa7b247af\") " pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.673379 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.673198    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/159a6253-516b-4444-81d9-d7daa7b247af-installation-pull-secrets\") pod \"image-registry-56b94bddb8-spm56\" (UID: \"159a6253-516b-4444-81d9-d7daa7b247af\") " pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.673800 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.673705    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/159a6253-516b-4444-81d9-d7daa7b247af-registry-tls\") pod \"image-registry-56b94bddb8-spm56\" (UID: \"159a6253-516b-4444-81d9-d7daa7b247af\") " pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.675278 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.675113    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/159a6253-516b-4444-81d9-d7daa7b247af-image-registry-private-configuration\") pod \"image-registry-56b94bddb8-spm56\" (UID: \"159a6253-516b-4444-81d9-d7daa7b247af\") " pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.679973 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.679938    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj75s\" (UniqueName: \"kubernetes.io/projected/159a6253-516b-4444-81d9-d7daa7b247af-kube-api-access-bj75s\") pod \"image-registry-56b94bddb8-spm56\" (UID: \"159a6253-516b-4444-81d9-d7daa7b247af\") " pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.681312 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.681276    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/159a6253-516b-4444-81d9-d7daa7b247af-bound-sa-token\") pod \"image-registry-56b94bddb8-spm56\" (UID: \"159a6253-516b-4444-81d9-d7daa7b247af\") " pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.703502 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.703462    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-6p4cp"
Sep 11 21:53:21.810754 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.810721    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:21.876346 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.875120    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-wplsx"]
Sep 11 21:53:21.909637 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.909593    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84c8f5cd7b-hd4bb"]
Sep 11 21:53:21.910479 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.910455    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bb5b4db8-vzdsr"]
Sep 11 21:53:21.913060 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:53:21.913034    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f5dbbfd_df55_47ec_b210_0dc2b22142e9.slice/crio-d5194c1bac7bd0276e0eab98e5d873a3fa2bba3a708767480b8e22da24f26326 WatchSource:0}: Error finding container d5194c1bac7bd0276e0eab98e5d873a3fa2bba3a708767480b8e22da24f26326: Status 404 returned error can't find the container with id d5194c1bac7bd0276e0eab98e5d873a3fa2bba3a708767480b8e22da24f26326
Sep 11 21:53:21.913456 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:53:21.913435    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c2350e_f194_4848_b674_08918601b698.slice/crio-b9f361aa2949556e39735078d3d124ccba61b5b9644edd456c62112dd4bd5692 WatchSource:0}: Error finding container b9f361aa2949556e39735078d3d124ccba61b5b9644edd456c62112dd4bd5692: Status 404 returned error can't find the container with id b9f361aa2949556e39735078d3d124ccba61b5b9644edd456c62112dd4bd5692
Sep 11 21:53:21.971727 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:21.971696    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-6p4cp"]
Sep 11 21:53:21.976938 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:53:21.976914    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c10e945_fa90_4b6a_b5a8_423c05d6db4d.slice/crio-58e7a046c48d03733a6d3d500114ca93c7f28bca76ad2de0381c115ba1a2c770 WatchSource:0}: Error finding container 58e7a046c48d03733a6d3d500114ca93c7f28bca76ad2de0381c115ba1a2c770: Status 404 returned error can't find the container with id 58e7a046c48d03733a6d3d500114ca93c7f28bca76ad2de0381c115ba1a2c770
Sep 11 21:53:22.100243 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:22.100060    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56b94bddb8-spm56"]
Sep 11 21:53:22.102527 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:53:22.102501    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod159a6253_516b_4444_81d9_d7daa7b247af.slice/crio-74be4ac7e80c296dec3fb70b991601668e1f9f50a4e6f1801d94820c650ee246 WatchSource:0}: Error finding container 74be4ac7e80c296dec3fb70b991601668e1f9f50a4e6f1801d94820c650ee246: Status 404 returned error can't find the container with id 74be4ac7e80c296dec3fb70b991601668e1f9f50a4e6f1801d94820c650ee246
Sep 11 21:53:22.597315 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:22.597271    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-6p4cp" event={"ID":"8c10e945-fa90-4b6a-b5a8-423c05d6db4d","Type":"ContainerStarted","Data":"58e7a046c48d03733a6d3d500114ca93c7f28bca76ad2de0381c115ba1a2c770"}
Sep 11 21:53:22.598962 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:22.598930    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84c8f5cd7b-hd4bb" event={"ID":"6f5dbbfd-df55-47ec-b210-0dc2b22142e9","Type":"ContainerStarted","Data":"d5194c1bac7bd0276e0eab98e5d873a3fa2bba3a708767480b8e22da24f26326"}
Sep 11 21:53:22.600256 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:22.600229    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bb5b4db8-vzdsr" event={"ID":"08c2350e-f194-4848-b674-08918601b698","Type":"ContainerStarted","Data":"b9f361aa2949556e39735078d3d124ccba61b5b9644edd456c62112dd4bd5692"}
Sep 11 21:53:22.602601 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:22.602278    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56b94bddb8-spm56" event={"ID":"159a6253-516b-4444-81d9-d7daa7b247af","Type":"ContainerStarted","Data":"ab0bfc2b185c64e2c9afb2b2e3bd45ced7d5cc3509b2bfc78daf5542cfee0f5a"}
Sep 11 21:53:22.602601 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:22.602320    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56b94bddb8-spm56" event={"ID":"159a6253-516b-4444-81d9-d7daa7b247af","Type":"ContainerStarted","Data":"74be4ac7e80c296dec3fb70b991601668e1f9f50a4e6f1801d94820c650ee246"}
Sep 11 21:53:22.602601 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:22.602455    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:22.603977 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:22.603947    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wplsx" event={"ID":"d10cc55c-a4d2-4c30-84d9-15ced74f00b3","Type":"ContainerStarted","Data":"9012a19ee239c7d68ca8bafd493dd4ca47c07199055b4f0935010a4077ceb60e"}
Sep 11 21:53:22.622037 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:22.621966    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56b94bddb8-spm56" podStartSLOduration=1.62194497 podStartE2EDuration="1.62194497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:22.621156518 +0000 UTC m=+55.796232905" watchObservedRunningTime="2026-09-11 21:53:22.62194497 +0000 UTC m=+55.797021341"
Sep 11 21:53:26.506956 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:26.506922    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lw6qf"
Sep 11 21:53:26.622304 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:26.622263    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wplsx" event={"ID":"d10cc55c-a4d2-4c30-84d9-15ced74f00b3","Type":"ContainerStarted","Data":"9e027ff5856e77fcc68d78dccc243149afbccb3a3cb4e33bca3ffd725e3b173d"}
Sep 11 21:53:26.624021 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:26.623987    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84c8f5cd7b-hd4bb" event={"ID":"6f5dbbfd-df55-47ec-b210-0dc2b22142e9","Type":"ContainerStarted","Data":"6001d4a3fe936f969d57d9f4eac9278a243089264d164bb7ccb3a9a5a6061ee7"}
Sep 11 21:53:26.638230 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:26.638170    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-wplsx" podStartSLOduration=1.540581628 podStartE2EDuration="5.638153601s" totalImagesPullingTime="4.097571973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:26.637023842 +0000 UTC m=+59.812100215" watchObservedRunningTime="2026-09-11 21:53:26.638153601 +0000 UTC m=+59.813229971"
Sep 11 21:53:26.652900 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:26.652846    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-84c8f5cd7b-hd4bb" podStartSLOduration=1.517459588 podStartE2EDuration="5.652824533s" totalImagesPullingTime="4.135364945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:26.652667127 +0000 UTC m=+59.827743499" watchObservedRunningTime="2026-09-11 21:53:26.652824533 +0000 UTC m=+59.827900905"
Sep 11 21:53:30.593678 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:30.593647    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rn9mr"
Sep 11 21:53:31.644492 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:31.644405    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bb5b4db8-vzdsr" event={"ID":"08c2350e-f194-4848-b674-08918601b698","Type":"ContainerStarted","Data":"d9217030f9120b5858c1f499f891f22f7f425de01461490abbc454ed6efc7378"}
Sep 11 21:53:31.644888 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:31.644668    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bb5b4db8-vzdsr"
Sep 11 21:53:31.646479 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:31.646422    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bb5b4db8-vzdsr"
Sep 11 21:53:31.664180 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:31.663271    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-59bb5b4db8-vzdsr" podStartSLOduration=1.330946811 podStartE2EDuration="10.663254186s" totalImagesPullingTime="9.332307375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:31.661965548 +0000 UTC m=+64.837041966" watchObservedRunningTime="2026-09-11 21:53:31.663254186 +0000 UTC m=+64.838330559"
Sep 11 21:53:33.079036 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:33.078994    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05523d8a-2c9a-47f3-ad9d-4c508546cd42-original-pull-secret\") pod \"global-pull-secret-syncer-fc5j4\" (UID: \"05523d8a-2c9a-47f3-ad9d-4c508546cd42\") " pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:53:33.079623 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:33.079119    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-metrics-certs\") pod \"network-metrics-daemon-n87wl\" (UID: \"92a0df5e-f5d1-4331-9ee2-7ed9916f00fd\") " pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:53:33.081980 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:33.081949    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 21:53:33.083197 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:33.083179    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 21:53:33.092267 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:33.092235    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92a0df5e-f5d1-4331-9ee2-7ed9916f00fd-metrics-certs\") pod \"network-metrics-daemon-n87wl\" (UID: \"92a0df5e-f5d1-4331-9ee2-7ed9916f00fd\") " pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:53:33.092452 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:33.092241    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/05523d8a-2c9a-47f3-ad9d-4c508546cd42-original-pull-secret\") pod \"global-pull-secret-syncer-fc5j4\" (UID: \"05523d8a-2c9a-47f3-ad9d-4c508546cd42\") " pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:53:33.180533 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:33.180487    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4vnl\" (UniqueName: \"kubernetes.io/projected/3752cc39-b860-4328-82d5-d67850cd906c-kube-api-access-b4vnl\") pod \"network-check-target-hxztv\" (UID: \"3752cc39-b860-4328-82d5-d67850cd906c\") " pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:53:33.183426 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:33.183395    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 21:53:33.194226 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:33.194202    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 21:53:33.204889 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:33.204859    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4vnl\" (UniqueName: \"kubernetes.io/projected/3752cc39-b860-4328-82d5-d67850cd906c-kube-api-access-b4vnl\") pod \"network-check-target-hxztv\" (UID: \"3752cc39-b860-4328-82d5-d67850cd906c\") " pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:53:33.244401 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:33.244364    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qsk4f\""
Sep 11 21:53:33.250474 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:33.250439    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4bfmc\""
Sep 11 21:53:33.251493 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:33.251470    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fc5j4"
Sep 11 21:53:33.252508 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:33.252489    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n87wl"
Sep 11 21:53:33.257778 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:33.257756    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:53:36.106481 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.106445    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68dbfd879c-8n65p"]
Sep 11 21:53:36.109801 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.109773    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:53:36.112701 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.112630    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 21:53:36.112701 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.112691    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-q495q\""
Sep 11 21:53:36.112893 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.112774    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 21:53:36.113918 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.113900    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 21:53:36.114025 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.113933    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 21:53:36.114190 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.114167    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 21:53:36.119266 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.119230    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 21:53:36.122698 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.122677    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68dbfd879c-8n65p"]
Sep 11 21:53:36.201492 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.201447    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f46fb9c-0e20-4009-98ba-317c8cd063f1-service-ca\") pod \"console-68dbfd879c-8n65p\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") " pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:53:36.201671 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.201528    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69th2\" (UniqueName: \"kubernetes.io/projected/3f46fb9c-0e20-4009-98ba-317c8cd063f1-kube-api-access-69th2\") pod \"console-68dbfd879c-8n65p\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") " pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:53:36.201735 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.201707    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f46fb9c-0e20-4009-98ba-317c8cd063f1-console-oauth-config\") pod \"console-68dbfd879c-8n65p\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") " pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:53:36.201796 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.201758    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f46fb9c-0e20-4009-98ba-317c8cd063f1-console-config\") pod \"console-68dbfd879c-8n65p\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") " pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:53:36.201796 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.201787    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f46fb9c-0e20-4009-98ba-317c8cd063f1-oauth-serving-cert\") pod \"console-68dbfd879c-8n65p\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") " pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:53:36.201900 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.201822    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f46fb9c-0e20-4009-98ba-317c8cd063f1-console-serving-cert\") pod \"console-68dbfd879c-8n65p\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") " pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:53:36.201954 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.201919    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f46fb9c-0e20-4009-98ba-317c8cd063f1-trusted-ca-bundle\") pod \"console-68dbfd879c-8n65p\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") " pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:53:36.302943 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.302900    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f46fb9c-0e20-4009-98ba-317c8cd063f1-console-oauth-config\") pod \"console-68dbfd879c-8n65p\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") " pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:53:36.302943 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.302951    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f46fb9c-0e20-4009-98ba-317c8cd063f1-console-config\") pod \"console-68dbfd879c-8n65p\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") " pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:53:36.303202 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.302979    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f46fb9c-0e20-4009-98ba-317c8cd063f1-oauth-serving-cert\") pod \"console-68dbfd879c-8n65p\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") " pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:53:36.303202 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.303010    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f46fb9c-0e20-4009-98ba-317c8cd063f1-console-serving-cert\") pod \"console-68dbfd879c-8n65p\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") " pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:53:36.303202 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.303066    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f46fb9c-0e20-4009-98ba-317c8cd063f1-trusted-ca-bundle\") pod \"console-68dbfd879c-8n65p\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") " pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:53:36.303202 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.303095    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f46fb9c-0e20-4009-98ba-317c8cd063f1-service-ca\") pod \"console-68dbfd879c-8n65p\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") " pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:53:36.303437 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.303291    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69th2\" (UniqueName: \"kubernetes.io/projected/3f46fb9c-0e20-4009-98ba-317c8cd063f1-kube-api-access-69th2\") pod \"console-68dbfd879c-8n65p\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") " pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:53:36.303971 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.303910    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f46fb9c-0e20-4009-98ba-317c8cd063f1-service-ca\") pod \"console-68dbfd879c-8n65p\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") " pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:53:36.303971 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.303916    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f46fb9c-0e20-4009-98ba-317c8cd063f1-console-config\") pod \"console-68dbfd879c-8n65p\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") " pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:53:36.304171 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.304141    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f46fb9c-0e20-4009-98ba-317c8cd063f1-trusted-ca-bundle\") pod \"console-68dbfd879c-8n65p\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") " pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:53:36.304405 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.304364    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f46fb9c-0e20-4009-98ba-317c8cd063f1-oauth-serving-cert\") pod \"console-68dbfd879c-8n65p\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") " pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:53:36.305841 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.305815    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f46fb9c-0e20-4009-98ba-317c8cd063f1-console-oauth-config\") pod \"console-68dbfd879c-8n65p\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") " pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:53:36.306159 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.306144    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f46fb9c-0e20-4009-98ba-317c8cd063f1-console-serving-cert\") pod \"console-68dbfd879c-8n65p\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") " pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:53:36.312170 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.312148    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69th2\" (UniqueName: \"kubernetes.io/projected/3f46fb9c-0e20-4009-98ba-317c8cd063f1-kube-api-access-69th2\") pod \"console-68dbfd879c-8n65p\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") " pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:53:36.420551 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:36.420513    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:53:37.693734 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.693698    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-fbsjn"]
Sep 11 21:53:37.729782 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.729752    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-fbsjn"]
Sep 11 21:53:37.729960 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.729810    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8zz2r"]
Sep 11 21:53:37.729960 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.729909    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fbsjn"
Sep 11 21:53:37.733844 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.733635    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 11 21:53:37.733844 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.733661    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 21:53:37.734202 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.734077    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 21:53:37.734202 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.734108    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 11 21:53:37.734372 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.734322    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 21:53:37.734579 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.734564    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-pl5qj\""
Sep 11 21:53:37.745749 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.745729    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.749766 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.749307    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 21:53:37.749766 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.749427    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-b2br2\""
Sep 11 21:53:37.749766 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.749531    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 21:53:37.749766 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.749633    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 21:53:37.815705 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.815675    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/194dd741-32d8-4ff6-be38-a16c389965fb-node-exporter-tls\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.815886 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.815721    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-fbsjn\" (UID: \"f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fbsjn"
Sep 11 21:53:37.815886 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.815747    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/194dd741-32d8-4ff6-be38-a16c389965fb-metrics-client-ca\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.815886 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.815767    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/194dd741-32d8-4ff6-be38-a16c389965fb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.815886 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.815789    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/194dd741-32d8-4ff6-be38-a16c389965fb-node-exporter-accelerators-collector-config\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.815886 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.815827    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-fbsjn\" (UID: \"f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fbsjn"
Sep 11 21:53:37.815886 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.815882    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/194dd741-32d8-4ff6-be38-a16c389965fb-node-exporter-textfile\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.816185 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.815906    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/194dd741-32d8-4ff6-be38-a16c389965fb-sys\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.816185 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.815936    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/194dd741-32d8-4ff6-be38-a16c389965fb-node-exporter-wtmp\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.816185 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.815965    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc4st\" (UniqueName: \"kubernetes.io/projected/194dd741-32d8-4ff6-be38-a16c389965fb-kube-api-access-mc4st\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.816185 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.815997    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpd4z\" (UniqueName: \"kubernetes.io/projected/f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6-kube-api-access-hpd4z\") pod \"openshift-state-metrics-67dcf8c979-fbsjn\" (UID: \"f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fbsjn"
Sep 11 21:53:37.816185 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.816025    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-fbsjn\" (UID: \"f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fbsjn"
Sep 11 21:53:37.816185 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.816051    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/194dd741-32d8-4ff6-be38-a16c389965fb-root\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.917103 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.917063    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/194dd741-32d8-4ff6-be38-a16c389965fb-node-exporter-textfile\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.917289 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.917108    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/194dd741-32d8-4ff6-be38-a16c389965fb-sys\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.917289 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.917178    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/194dd741-32d8-4ff6-be38-a16c389965fb-sys\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.917442 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.917289    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/194dd741-32d8-4ff6-be38-a16c389965fb-node-exporter-wtmp\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.917442 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.917344    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc4st\" (UniqueName: \"kubernetes.io/projected/194dd741-32d8-4ff6-be38-a16c389965fb-kube-api-access-mc4st\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.917442 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.917378    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpd4z\" (UniqueName: \"kubernetes.io/projected/f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6-kube-api-access-hpd4z\") pod \"openshift-state-metrics-67dcf8c979-fbsjn\" (UID: \"f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fbsjn"
Sep 11 21:53:37.917442 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.917411    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-fbsjn\" (UID: \"f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fbsjn"
Sep 11 21:53:37.917442 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.917437    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/194dd741-32d8-4ff6-be38-a16c389965fb-root\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.917698 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.917470    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/194dd741-32d8-4ff6-be38-a16c389965fb-node-exporter-tls\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.917698 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.917478    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/194dd741-32d8-4ff6-be38-a16c389965fb-node-exporter-textfile\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.917698 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.917505    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-fbsjn\" (UID: \"f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fbsjn"
Sep 11 21:53:37.917698 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.917533    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/194dd741-32d8-4ff6-be38-a16c389965fb-metrics-client-ca\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.917698 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.917557    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/194dd741-32d8-4ff6-be38-a16c389965fb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.917698 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.917581    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/194dd741-32d8-4ff6-be38-a16c389965fb-node-exporter-accelerators-collector-config\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.917698 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.917625    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-fbsjn\" (UID: \"f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fbsjn"
Sep 11 21:53:37.918072 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:37.917754    2587 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 11 21:53:37.918072 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:37.917818    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6-openshift-state-metrics-tls podName:f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6 nodeName:}" failed. No retries permitted until 2026-09-11 21:53:38.417797378 +0000 UTC m=+71.592873742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-fbsjn" (UID: "f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6") : secret "openshift-state-metrics-tls" not found
Sep 11 21:53:37.918072 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.917913    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/194dd741-32d8-4ff6-be38-a16c389965fb-node-exporter-wtmp\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.918533 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:37.918514    2587 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 21:53:37.919944 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:53:37.919709    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/194dd741-32d8-4ff6-be38-a16c389965fb-node-exporter-tls podName:194dd741-32d8-4ff6-be38-a16c389965fb nodeName:}" failed. No retries permitted until 2026-09-11 21:53:38.419683978 +0000 UTC m=+71.594760359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/194dd741-32d8-4ff6-be38-a16c389965fb-node-exporter-tls") pod "node-exporter-8zz2r" (UID: "194dd741-32d8-4ff6-be38-a16c389965fb") : secret "node-exporter-tls" not found
Sep 11 21:53:37.934249 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.923610    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/194dd741-32d8-4ff6-be38-a16c389965fb-metrics-client-ca\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.934249 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.924096    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-fbsjn\" (UID: \"f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fbsjn"
Sep 11 21:53:37.934249 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.927157    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpd4z\" (UniqueName: \"kubernetes.io/projected/f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6-kube-api-access-hpd4z\") pod \"openshift-state-metrics-67dcf8c979-fbsjn\" (UID: \"f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fbsjn"
Sep 11 21:53:37.934249 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.928861    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/194dd741-32d8-4ff6-be38-a16c389965fb-root\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.934249 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.929926    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc4st\" (UniqueName: \"kubernetes.io/projected/194dd741-32d8-4ff6-be38-a16c389965fb-kube-api-access-mc4st\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.934249 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.930577    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-fbsjn\" (UID: \"f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fbsjn"
Sep 11 21:53:37.934643 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.934312    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/194dd741-32d8-4ff6-be38-a16c389965fb-node-exporter-accelerators-collector-config\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:37.934975 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:37.934927    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/194dd741-32d8-4ff6-be38-a16c389965fb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:38.424350 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:38.424292    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/194dd741-32d8-4ff6-be38-a16c389965fb-node-exporter-tls\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:38.424556 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:38.424385    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-fbsjn\" (UID: \"f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fbsjn"
Sep 11 21:53:38.426989 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:38.426959    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/194dd741-32d8-4ff6-be38-a16c389965fb-node-exporter-tls\") pod \"node-exporter-8zz2r\" (UID: \"194dd741-32d8-4ff6-be38-a16c389965fb\") " pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:38.427115 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:38.427035    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-fbsjn\" (UID: \"f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fbsjn"
Sep 11 21:53:38.644232 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:38.644194    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fbsjn"
Sep 11 21:53:38.659463 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:38.659437    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8zz2r"
Sep 11 21:53:39.308084 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:39.307913    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n87wl"]
Sep 11 21:53:39.310730 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:53:39.310698    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a0df5e_f5d1_4331_9ee2_7ed9916f00fd.slice/crio-5ee3b485cc1313c5b6a1eeb2453d989a97a8ebf3924891d63d6e106b3bbe5d22 WatchSource:0}: Error finding container 5ee3b485cc1313c5b6a1eeb2453d989a97a8ebf3924891d63d6e106b3bbe5d22: Status 404 returned error can't find the container with id 5ee3b485cc1313c5b6a1eeb2453d989a97a8ebf3924891d63d6e106b3bbe5d22
Sep 11 21:53:39.311940 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:39.311905    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-fbsjn"]
Sep 11 21:53:39.313477 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:39.313453    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68dbfd879c-8n65p"]
Sep 11 21:53:39.315106 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:53:39.315085    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f46fb9c_0e20_4009_98ba_317c8cd063f1.slice/crio-4b61296a374c073c8a549174d4acebdf0ad3cd8bbf2b0ab7e7af54e7f5ce831f WatchSource:0}: Error finding container 4b61296a374c073c8a549174d4acebdf0ad3cd8bbf2b0ab7e7af54e7f5ce831f: Status 404 returned error can't find the container with id 4b61296a374c073c8a549174d4acebdf0ad3cd8bbf2b0ab7e7af54e7f5ce831f
Sep 11 21:53:39.316019 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:53:39.315969    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2fe79e0_afd9_4e59_91f8_76a8d0bbc2a6.slice/crio-16d4200b00a912c9ad6a7afca85580d1931f385d48b724e914156447a127d5a3 WatchSource:0}: Error finding container 16d4200b00a912c9ad6a7afca85580d1931f385d48b724e914156447a127d5a3: Status 404 returned error can't find the container with id 16d4200b00a912c9ad6a7afca85580d1931f385d48b724e914156447a127d5a3
Sep 11 21:53:39.364353 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:39.364065    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fc5j4"]
Sep 11 21:53:39.367676 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:39.367645    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hxztv"]
Sep 11 21:53:39.368201 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:53:39.368124    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05523d8a_2c9a_47f3_ad9d_4c508546cd42.slice/crio-1c4a5e213a9fcc50e8f65399c5cce36baaaa0aae2b48c799954000b93d98fd82 WatchSource:0}: Error finding container 1c4a5e213a9fcc50e8f65399c5cce36baaaa0aae2b48c799954000b93d98fd82: Status 404 returned error can't find the container with id 1c4a5e213a9fcc50e8f65399c5cce36baaaa0aae2b48c799954000b93d98fd82
Sep 11 21:53:39.370245 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:53:39.370152    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3752cc39_b860_4328_82d5_d67850cd906c.slice/crio-53777b009ebd505c6b674ee8065bf8cd69da6d9fb6bd42d0324db81656fff7b1 WatchSource:0}: Error finding container 53777b009ebd505c6b674ee8065bf8cd69da6d9fb6bd42d0324db81656fff7b1: Status 404 returned error can't find the container with id 53777b009ebd505c6b674ee8065bf8cd69da6d9fb6bd42d0324db81656fff7b1
Sep 11 21:53:39.677710 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:39.677638    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-6p4cp" event={"ID":"8c10e945-fa90-4b6a-b5a8-423c05d6db4d","Type":"ContainerStarted","Data":"6671ad2ff97910d46d69513c38883a818c24d9f01d986384028bb80573be3eeb"}
Sep 11 21:53:39.678363 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:39.678322    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-6p4cp"
Sep 11 21:53:39.680273 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:39.680010    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8zz2r" event={"ID":"194dd741-32d8-4ff6-be38-a16c389965fb","Type":"ContainerStarted","Data":"8d954021761a7127f5535d13dce4f7d3738e9c2b9e0440d44686d035974f5d8c"}
Sep 11 21:53:39.683095 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:39.682943    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fbsjn" event={"ID":"f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6","Type":"ContainerStarted","Data":"ac4477234fb99b1267e08f50dac0e33dd1405f135ca8ce0bc4a2427185328080"}
Sep 11 21:53:39.683095 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:39.683002    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fbsjn" event={"ID":"f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6","Type":"ContainerStarted","Data":"2f3392580e586ae4deb013a10b26d7e09c2afce1e30bd6113a250571e327cce6"}
Sep 11 21:53:39.683095 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:39.683033    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fbsjn" event={"ID":"f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6","Type":"ContainerStarted","Data":"16d4200b00a912c9ad6a7afca85580d1931f385d48b724e914156447a127d5a3"}
Sep 11 21:53:39.689250 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:39.689134    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68dbfd879c-8n65p" event={"ID":"3f46fb9c-0e20-4009-98ba-317c8cd063f1","Type":"ContainerStarted","Data":"4b61296a374c073c8a549174d4acebdf0ad3cd8bbf2b0ab7e7af54e7f5ce831f"}
Sep 11 21:53:39.689250 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:39.689220    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-6p4cp"
Sep 11 21:53:39.691768 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:39.691628    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hxztv" event={"ID":"3752cc39-b860-4328-82d5-d67850cd906c","Type":"ContainerStarted","Data":"53777b009ebd505c6b674ee8065bf8cd69da6d9fb6bd42d0324db81656fff7b1"}
Sep 11 21:53:39.693359 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:39.693290    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fc5j4" event={"ID":"05523d8a-2c9a-47f3-ad9d-4c508546cd42","Type":"ContainerStarted","Data":"1c4a5e213a9fcc50e8f65399c5cce36baaaa0aae2b48c799954000b93d98fd82"}
Sep 11 21:53:39.697447 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:39.696826    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-6p4cp" podStartSLOduration=1.532905035 podStartE2EDuration="18.696807088s" totalImagesPullingTime="17.163902053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:39.695349573 +0000 UTC m=+72.870425937" watchObservedRunningTime="2026-09-11 21:53:39.696807088 +0000 UTC m=+72.871883460"
Sep 11 21:53:39.698196 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:39.698115    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n87wl" event={"ID":"92a0df5e-f5d1-4331-9ee2-7ed9916f00fd","Type":"ContainerStarted","Data":"5ee3b485cc1313c5b6a1eeb2453d989a97a8ebf3924891d63d6e106b3bbe5d22"}
Sep 11 21:53:42.108403 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.108367    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5b7987cc77-2xgzc"]
Sep 11 21:53:42.141271 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.141220    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b7987cc77-2xgzc"]
Sep 11 21:53:42.141439 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.141374    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:53:42.145748 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.145724    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 11 21:53:42.146001 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.145983    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5t2hsh8snjbc7\""
Sep 11 21:53:42.146200 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.146181    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 11 21:53:42.146417 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.146401    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 11 21:53:42.146611 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.146593    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-h6w5x\""
Sep 11 21:53:42.146843 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.146823    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 21:53:42.263553 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.263503    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9380b76d-3c91-414b-a9db-5ec96971b603-client-ca-bundle\") pod \"metrics-server-5b7987cc77-2xgzc\" (UID: \"9380b76d-3c91-414b-a9db-5ec96971b603\") " pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:53:42.263733 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.263576    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9380b76d-3c91-414b-a9db-5ec96971b603-audit-log\") pod \"metrics-server-5b7987cc77-2xgzc\" (UID: \"9380b76d-3c91-414b-a9db-5ec96971b603\") " pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:53:42.263733 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.263621    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9380b76d-3c91-414b-a9db-5ec96971b603-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b7987cc77-2xgzc\" (UID: \"9380b76d-3c91-414b-a9db-5ec96971b603\") " pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:53:42.263733 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.263657    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqqp6\" (UniqueName: \"kubernetes.io/projected/9380b76d-3c91-414b-a9db-5ec96971b603-kube-api-access-wqqp6\") pod \"metrics-server-5b7987cc77-2xgzc\" (UID: \"9380b76d-3c91-414b-a9db-5ec96971b603\") " pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:53:42.263733 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.263717    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9380b76d-3c91-414b-a9db-5ec96971b603-metrics-server-audit-profiles\") pod \"metrics-server-5b7987cc77-2xgzc\" (UID: \"9380b76d-3c91-414b-a9db-5ec96971b603\") " pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:53:42.263961 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.263762    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9380b76d-3c91-414b-a9db-5ec96971b603-secret-metrics-server-client-certs\") pod \"metrics-server-5b7987cc77-2xgzc\" (UID: \"9380b76d-3c91-414b-a9db-5ec96971b603\") " pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:53:42.263961 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.263786    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9380b76d-3c91-414b-a9db-5ec96971b603-secret-metrics-server-tls\") pod \"metrics-server-5b7987cc77-2xgzc\" (UID: \"9380b76d-3c91-414b-a9db-5ec96971b603\") " pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:53:42.365168 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.365094    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9380b76d-3c91-414b-a9db-5ec96971b603-audit-log\") pod \"metrics-server-5b7987cc77-2xgzc\" (UID: \"9380b76d-3c91-414b-a9db-5ec96971b603\") " pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:53:42.365168 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.365164    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9380b76d-3c91-414b-a9db-5ec96971b603-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b7987cc77-2xgzc\" (UID: \"9380b76d-3c91-414b-a9db-5ec96971b603\") " pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:53:42.365419 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.365231    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqqp6\" (UniqueName: \"kubernetes.io/projected/9380b76d-3c91-414b-a9db-5ec96971b603-kube-api-access-wqqp6\") pod \"metrics-server-5b7987cc77-2xgzc\" (UID: \"9380b76d-3c91-414b-a9db-5ec96971b603\") " pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:53:42.365419 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.365257    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9380b76d-3c91-414b-a9db-5ec96971b603-metrics-server-audit-profiles\") pod \"metrics-server-5b7987cc77-2xgzc\" (UID: \"9380b76d-3c91-414b-a9db-5ec96971b603\") " pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:53:42.365419 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.365301    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9380b76d-3c91-414b-a9db-5ec96971b603-secret-metrics-server-client-certs\") pod \"metrics-server-5b7987cc77-2xgzc\" (UID: \"9380b76d-3c91-414b-a9db-5ec96971b603\") " pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:53:42.368493 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.366059    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9380b76d-3c91-414b-a9db-5ec96971b603-audit-log\") pod \"metrics-server-5b7987cc77-2xgzc\" (UID: \"9380b76d-3c91-414b-a9db-5ec96971b603\") " pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:53:42.368493 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.366731    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9380b76d-3c91-414b-a9db-5ec96971b603-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b7987cc77-2xgzc\" (UID: \"9380b76d-3c91-414b-a9db-5ec96971b603\") " pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:53:42.368493 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.367655    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9380b76d-3c91-414b-a9db-5ec96971b603-metrics-server-audit-profiles\") pod \"metrics-server-5b7987cc77-2xgzc\" (UID: \"9380b76d-3c91-414b-a9db-5ec96971b603\") " pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:53:42.369293 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.365343    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9380b76d-3c91-414b-a9db-5ec96971b603-secret-metrics-server-tls\") pod \"metrics-server-5b7987cc77-2xgzc\" (UID: \"9380b76d-3c91-414b-a9db-5ec96971b603\") " pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:53:42.369293 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.368873    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9380b76d-3c91-414b-a9db-5ec96971b603-client-ca-bundle\") pod \"metrics-server-5b7987cc77-2xgzc\" (UID: \"9380b76d-3c91-414b-a9db-5ec96971b603\") " pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:53:42.371648 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.371622    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9380b76d-3c91-414b-a9db-5ec96971b603-secret-metrics-server-tls\") pod \"metrics-server-5b7987cc77-2xgzc\" (UID: \"9380b76d-3c91-414b-a9db-5ec96971b603\") " pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:53:42.374610 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.374544    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9380b76d-3c91-414b-a9db-5ec96971b603-client-ca-bundle\") pod \"metrics-server-5b7987cc77-2xgzc\" (UID: \"9380b76d-3c91-414b-a9db-5ec96971b603\") " pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:53:42.380544 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.380502    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9380b76d-3c91-414b-a9db-5ec96971b603-secret-metrics-server-client-certs\") pod \"metrics-server-5b7987cc77-2xgzc\" (UID: \"9380b76d-3c91-414b-a9db-5ec96971b603\") " pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:53:42.383197 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.383137    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqqp6\" (UniqueName: \"kubernetes.io/projected/9380b76d-3c91-414b-a9db-5ec96971b603-kube-api-access-wqqp6\") pod \"metrics-server-5b7987cc77-2xgzc\" (UID: \"9380b76d-3c91-414b-a9db-5ec96971b603\") " pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:53:42.456815 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.456528    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:53:42.534204 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.534174    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68dbfd879c-8n65p"]
Sep 11 21:53:42.915125 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.915092    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"]
Sep 11 21:53:42.939496 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.939007    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:42.939644 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.939588    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"]
Sep 11 21:53:42.946181 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.946151    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 11 21:53:42.946439 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.946420    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rk5jm\""
Sep 11 21:53:42.946643 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.946624    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 11 21:53:42.946873 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.946854    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 11 21:53:42.947253 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.947232    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 11 21:53:42.947555 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.947526    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 11 21:53:42.952867 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:42.952841    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 11 21:53:43.072516 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.070693    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b7987cc77-2xgzc"]
Sep 11 21:53:43.078955 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.078403    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3de3479-780f-4ea1-9eb4-8cc83cfa8deb-serving-certs-ca-bundle\") pod \"telemeter-client-5cf76b897f-jbmhr\" (UID: \"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb\") " pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:43.078955 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.078444    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3de3479-780f-4ea1-9eb4-8cc83cfa8deb-metrics-client-ca\") pod \"telemeter-client-5cf76b897f-jbmhr\" (UID: \"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb\") " pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:43.078955 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.078531    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c3de3479-780f-4ea1-9eb4-8cc83cfa8deb-telemeter-client-tls\") pod \"telemeter-client-5cf76b897f-jbmhr\" (UID: \"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb\") " pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:43.078955 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.078593    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c3de3479-780f-4ea1-9eb4-8cc83cfa8deb-secret-telemeter-client\") pod \"telemeter-client-5cf76b897f-jbmhr\" (UID: \"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb\") " pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:43.078955 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.078628    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb62v\" (UniqueName: \"kubernetes.io/projected/c3de3479-780f-4ea1-9eb4-8cc83cfa8deb-kube-api-access-pb62v\") pod \"telemeter-client-5cf76b897f-jbmhr\" (UID: \"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb\") " pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:43.078955 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.078671    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3de3479-780f-4ea1-9eb4-8cc83cfa8deb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5cf76b897f-jbmhr\" (UID: \"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb\") " pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:43.078955 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.078737    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3de3479-780f-4ea1-9eb4-8cc83cfa8deb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5cf76b897f-jbmhr\" (UID: \"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb\") " pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:43.078955 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.078797    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c3de3479-780f-4ea1-9eb4-8cc83cfa8deb-federate-client-tls\") pod \"telemeter-client-5cf76b897f-jbmhr\" (UID: \"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb\") " pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:43.114116 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:53:43.112239    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9380b76d_3c91_414b_a9db_5ec96971b603.slice/crio-e1a8c139f57e01ad92d839a275fd1175368e5b2bcd97c69a440de09b7e033531 WatchSource:0}: Error finding container e1a8c139f57e01ad92d839a275fd1175368e5b2bcd97c69a440de09b7e033531: Status 404 returned error can't find the container with id e1a8c139f57e01ad92d839a275fd1175368e5b2bcd97c69a440de09b7e033531
Sep 11 21:53:43.182983 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.179939    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3de3479-780f-4ea1-9eb4-8cc83cfa8deb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5cf76b897f-jbmhr\" (UID: \"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb\") " pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:43.182983 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.179990    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3de3479-780f-4ea1-9eb4-8cc83cfa8deb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5cf76b897f-jbmhr\" (UID: \"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb\") " pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:43.182983 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.180061    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c3de3479-780f-4ea1-9eb4-8cc83cfa8deb-federate-client-tls\") pod \"telemeter-client-5cf76b897f-jbmhr\" (UID: \"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb\") " pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:43.182983 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.180112    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3de3479-780f-4ea1-9eb4-8cc83cfa8deb-serving-certs-ca-bundle\") pod \"telemeter-client-5cf76b897f-jbmhr\" (UID: \"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb\") " pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:43.182983 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.180139    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3de3479-780f-4ea1-9eb4-8cc83cfa8deb-metrics-client-ca\") pod \"telemeter-client-5cf76b897f-jbmhr\" (UID: \"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb\") " pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:43.182983 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.180168    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c3de3479-780f-4ea1-9eb4-8cc83cfa8deb-telemeter-client-tls\") pod \"telemeter-client-5cf76b897f-jbmhr\" (UID: \"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb\") " pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:43.182983 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.180228    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c3de3479-780f-4ea1-9eb4-8cc83cfa8deb-secret-telemeter-client\") pod \"telemeter-client-5cf76b897f-jbmhr\" (UID: \"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb\") " pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:43.182983 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.180258    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb62v\" (UniqueName: \"kubernetes.io/projected/c3de3479-780f-4ea1-9eb4-8cc83cfa8deb-kube-api-access-pb62v\") pod \"telemeter-client-5cf76b897f-jbmhr\" (UID: \"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb\") " pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:43.182983 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.181724    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3de3479-780f-4ea1-9eb4-8cc83cfa8deb-serving-certs-ca-bundle\") pod \"telemeter-client-5cf76b897f-jbmhr\" (UID: \"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb\") " pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:43.182983 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.182668    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3de3479-780f-4ea1-9eb4-8cc83cfa8deb-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5cf76b897f-jbmhr\" (UID: \"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb\") " pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:43.184746 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.184678    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3de3479-780f-4ea1-9eb4-8cc83cfa8deb-metrics-client-ca\") pod \"telemeter-client-5cf76b897f-jbmhr\" (UID: \"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb\") " pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:43.186872 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.186821    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c3de3479-780f-4ea1-9eb4-8cc83cfa8deb-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5cf76b897f-jbmhr\" (UID: \"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb\") " pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:43.187629 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.187602    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c3de3479-780f-4ea1-9eb4-8cc83cfa8deb-federate-client-tls\") pod \"telemeter-client-5cf76b897f-jbmhr\" (UID: \"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb\") " pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:43.189355 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.189289    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c3de3479-780f-4ea1-9eb4-8cc83cfa8deb-secret-telemeter-client\") pod \"telemeter-client-5cf76b897f-jbmhr\" (UID: \"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb\") " pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:43.190166 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.190121    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb62v\" (UniqueName: \"kubernetes.io/projected/c3de3479-780f-4ea1-9eb4-8cc83cfa8deb-kube-api-access-pb62v\") pod \"telemeter-client-5cf76b897f-jbmhr\" (UID: \"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb\") " pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:43.190376 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.190315    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c3de3479-780f-4ea1-9eb4-8cc83cfa8deb-telemeter-client-tls\") pod \"telemeter-client-5cf76b897f-jbmhr\" (UID: \"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb\") " pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:43.265484 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.265255    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"
Sep 11 21:53:43.603643 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.603506    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5cf76b897f-jbmhr"]
Sep 11 21:53:43.615159 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.615107    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56b94bddb8-spm56"
Sep 11 21:53:43.739810 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.739774    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fbsjn" event={"ID":"f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6","Type":"ContainerStarted","Data":"e59a26a81d5c97ed269b7086b28eabb960bd71eb46b0885d3a02a95a1e69c305"}
Sep 11 21:53:43.750704 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.750671    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc" event={"ID":"9380b76d-3c91-414b-a9db-5ec96971b603","Type":"ContainerStarted","Data":"e1a8c139f57e01ad92d839a275fd1175368e5b2bcd97c69a440de09b7e033531"}
Sep 11 21:53:43.754966 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.754933    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n87wl" event={"ID":"92a0df5e-f5d1-4331-9ee2-7ed9916f00fd","Type":"ContainerStarted","Data":"7d74698528e86faba0bdd66d29569e635fd4ef7bfed35ac62a9adaf71b526ebf"}
Sep 11 21:53:43.755108 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.754990    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n87wl" event={"ID":"92a0df5e-f5d1-4331-9ee2-7ed9916f00fd","Type":"ContainerStarted","Data":"590b7913fc96c32f45a204730162c23c94f1bdd42204b4082470d51f88c1803a"}
Sep 11 21:53:43.758067 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.758038    2587 generic.go:356] "Generic (PLEG): container finished" podID="194dd741-32d8-4ff6-be38-a16c389965fb" containerID="7f313c6ae6f96f36f2c51375a9a56da3f190dd7ce19edbd77d3e00845cd09987" exitCode=0
Sep 11 21:53:43.758172 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.758093    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8zz2r" event={"ID":"194dd741-32d8-4ff6-be38-a16c389965fb","Type":"ContainerDied","Data":"7f313c6ae6f96f36f2c51375a9a56da3f190dd7ce19edbd77d3e00845cd09987"}
Sep 11 21:53:43.762169 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.762117    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-fbsjn" podStartSLOduration=3.472084658 podStartE2EDuration="6.76208548s" totalImagesPullingTime="3.290000822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:43.76131855 +0000 UTC m=+76.936394945" watchObservedRunningTime="2026-09-11 21:53:43.76208548 +0000 UTC m=+76.937161841"
Sep 11 21:53:43.781948 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:43.781891    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-n87wl" podStartSLOduration=73.340278874 podStartE2EDuration="1m16.781869205s" totalImagesPullingTime="3.441590331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:43.780843923 +0000 UTC m=+76.955920308" watchObservedRunningTime="2026-09-11 21:53:43.781869205 +0000 UTC m=+76.956945577"
Sep 11 21:53:45.419558 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:53:45.419509    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3de3479_780f_4ea1_9eb4_8cc83cfa8deb.slice/crio-ce3a3c3bcfcfd0ce10edb20d4b8cdffac6fdba3fd5d4f245c6bd5417a2cf843d WatchSource:0}: Error finding container ce3a3c3bcfcfd0ce10edb20d4b8cdffac6fdba3fd5d4f245c6bd5417a2cf843d: Status 404 returned error can't find the container with id ce3a3c3bcfcfd0ce10edb20d4b8cdffac6fdba3fd5d4f245c6bd5417a2cf843d
Sep 11 21:53:45.770419 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:45.770311    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr" event={"ID":"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb","Type":"ContainerStarted","Data":"ce3a3c3bcfcfd0ce10edb20d4b8cdffac6fdba3fd5d4f245c6bd5417a2cf843d"}
Sep 11 21:53:49.792538 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:49.792441    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68dbfd879c-8n65p" event={"ID":"3f46fb9c-0e20-4009-98ba-317c8cd063f1","Type":"ContainerStarted","Data":"9ae0158870dc614c3e2828b522f61718622aff94c76cd5cf6d898968cce65790"}
Sep 11 21:53:49.794654 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:49.794621    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc" event={"ID":"9380b76d-3c91-414b-a9db-5ec96971b603","Type":"ContainerStarted","Data":"e9a4017faaa89023fb55af8820b92689a8b600bcdaba8f4c77907d801838132b"}
Sep 11 21:53:49.796953 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:49.796566    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hxztv" event={"ID":"3752cc39-b860-4328-82d5-d67850cd906c","Type":"ContainerStarted","Data":"dedee5d8350c2698b0f0a838c350b13c355a89a3ff6d8529e36f0db6e3ac1acf"}
Sep 11 21:53:49.796953 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:49.796636    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:53:49.799135 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:49.799108    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fc5j4" event={"ID":"05523d8a-2c9a-47f3-ad9d-4c508546cd42","Type":"ContainerStarted","Data":"8b5e8daa58c4bd774fc43f550095b887da335415d996518d5804ec993015cfcc"}
Sep 11 21:53:49.802211 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:49.802188    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8zz2r" event={"ID":"194dd741-32d8-4ff6-be38-a16c389965fb","Type":"ContainerStarted","Data":"5d69f90939c447ede1353384a6e0cf50aeb9aca40967573e7a48664ce83b062c"}
Sep 11 21:53:49.802319 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:49.802232    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8zz2r" event={"ID":"194dd741-32d8-4ff6-be38-a16c389965fb","Type":"ContainerStarted","Data":"4187d222373f6dab6ca8ad45021effc5944ece4c2ad60119ae2c1a64df17a132"}
Sep 11 21:53:49.814700 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:49.814659    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68dbfd879c-8n65p" podStartSLOduration=4.042959374 podStartE2EDuration="13.814645371s" totalImagesPullingTime="9.771685997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:53:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:49.81319917 +0000 UTC m=+82.988275541" watchObservedRunningTime="2026-09-11 21:53:49.814645371 +0000 UTC m=+82.989721740"
Sep 11 21:53:49.830553 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:49.830509    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8zz2r" podStartSLOduration=9.034698407 podStartE2EDuration="12.830491332s" totalImagesPullingTime="3.668433758s" totalInitContainerRuntime="127.359167ms" isStatefulPod=true podCreationTimestamp="2026-09-11 21:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:49.830061464 +0000 UTC m=+83.005137860" watchObservedRunningTime="2026-09-11 21:53:49.830491332 +0000 UTC m=+83.005567703"
Sep 11 21:53:49.848934 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:49.848884    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc" podStartSLOduration=1.566146656 podStartE2EDuration="7.848871722s" totalImagesPullingTime="6.282725066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:49.84766645 +0000 UTC m=+83.022742846" watchObservedRunningTime="2026-09-11 21:53:49.848871722 +0000 UTC m=+83.023948091"
Sep 11 21:53:49.863174 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:49.863111    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fc5j4" podStartSLOduration=73.144681848 podStartE2EDuration="1m22.863090692s" totalImagesPullingTime="9.718408844s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 21:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:49.862440225 +0000 UTC m=+83.037516621" watchObservedRunningTime="2026-09-11 21:53:49.863090692 +0000 UTC m=+83.038167070"
Sep 11 21:53:49.878226 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:49.878175    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hxztv" podStartSLOduration=73.161104127 podStartE2EDuration="1m22.878161113s" totalImagesPullingTime="9.717056986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:49.877482249 +0000 UTC m=+83.052558640" watchObservedRunningTime="2026-09-11 21:53:49.878161113 +0000 UTC m=+83.053237482"
Sep 11 21:53:50.807442 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:50.807356    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr" event={"ID":"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb","Type":"ContainerStarted","Data":"b534e9f19e1bd0ed4f31d10b2b6bb676249af9c64c05e4ecf3be5712cd2c48eb"}
Sep 11 21:53:52.817004 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:52.816972    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr" event={"ID":"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb","Type":"ContainerStarted","Data":"06a6a1f2def78b32f7bcd751d1ef601acdd7caf96b42e8736f29cc5dbc074fa4"}
Sep 11 21:53:52.817407 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:52.817023    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr" event={"ID":"c3de3479-780f-4ea1-9eb4-8cc83cfa8deb","Type":"ContainerStarted","Data":"65c20f08cbc9dc353b06b5b404963f1f30975ad4faf9b21584467834a2317269"}
Sep 11 21:53:52.842480 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:52.842422    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5cf76b897f-jbmhr" podStartSLOduration=4.338202835 podStartE2EDuration="10.842381378s" totalImagesPullingTime="6.504178543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:53:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:53:52.840925099 +0000 UTC m=+86.016001502" watchObservedRunningTime="2026-09-11 21:53:52.842381378 +0000 UTC m=+86.017457749"
Sep 11 21:53:56.420927 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:53:56.420892    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:54:02.457887 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:02.457845    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:54:02.458370 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:02.457908    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:54:14.814405 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:14.814353    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68dbfd879c-8n65p" podUID="3f46fb9c-0e20-4009-98ba-317c8cd063f1" containerName="console" containerID="cri-o://9ae0158870dc614c3e2828b522f61718622aff94c76cd5cf6d898968cce65790" gracePeriod=15
Sep 11 21:54:15.066768 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.066705    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:54:15.147463 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.147428    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f46fb9c-0e20-4009-98ba-317c8cd063f1-service-ca\") pod \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") "
Sep 11 21:54:15.147463 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.147471    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69th2\" (UniqueName: \"kubernetes.io/projected/3f46fb9c-0e20-4009-98ba-317c8cd063f1-kube-api-access-69th2\") pod \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") "
Sep 11 21:54:15.147700 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.147492    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f46fb9c-0e20-4009-98ba-317c8cd063f1-trusted-ca-bundle\") pod \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") "
Sep 11 21:54:15.147700 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.147517    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f46fb9c-0e20-4009-98ba-317c8cd063f1-console-serving-cert\") pod \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") "
Sep 11 21:54:15.147700 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.147650    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f46fb9c-0e20-4009-98ba-317c8cd063f1-oauth-serving-cert\") pod \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") "
Sep 11 21:54:15.147834 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.147712    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f46fb9c-0e20-4009-98ba-317c8cd063f1-console-oauth-config\") pod \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") "
Sep 11 21:54:15.147834 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.147742    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f46fb9c-0e20-4009-98ba-317c8cd063f1-console-config\") pod \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\" (UID: \"3f46fb9c-0e20-4009-98ba-317c8cd063f1\") "
Sep 11 21:54:15.148036 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.148006    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f46fb9c-0e20-4009-98ba-317c8cd063f1-service-ca" (OuterVolumeSpecName: "service-ca") pod "3f46fb9c-0e20-4009-98ba-317c8cd063f1" (UID: "3f46fb9c-0e20-4009-98ba-317c8cd063f1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:54:15.148036 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.148017    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f46fb9c-0e20-4009-98ba-317c8cd063f1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3f46fb9c-0e20-4009-98ba-317c8cd063f1" (UID: "3f46fb9c-0e20-4009-98ba-317c8cd063f1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:54:15.148217 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.148193    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f46fb9c-0e20-4009-98ba-317c8cd063f1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3f46fb9c-0e20-4009-98ba-317c8cd063f1" (UID: "3f46fb9c-0e20-4009-98ba-317c8cd063f1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:54:15.148320 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.148299    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f46fb9c-0e20-4009-98ba-317c8cd063f1-console-config" (OuterVolumeSpecName: "console-config") pod "3f46fb9c-0e20-4009-98ba-317c8cd063f1" (UID: "3f46fb9c-0e20-4009-98ba-317c8cd063f1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:54:15.149821 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.149794    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f46fb9c-0e20-4009-98ba-317c8cd063f1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3f46fb9c-0e20-4009-98ba-317c8cd063f1" (UID: "3f46fb9c-0e20-4009-98ba-317c8cd063f1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:54:15.149914 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.149901    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f46fb9c-0e20-4009-98ba-317c8cd063f1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3f46fb9c-0e20-4009-98ba-317c8cd063f1" (UID: "3f46fb9c-0e20-4009-98ba-317c8cd063f1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:54:15.149970 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.149930    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f46fb9c-0e20-4009-98ba-317c8cd063f1-kube-api-access-69th2" (OuterVolumeSpecName: "kube-api-access-69th2") pod "3f46fb9c-0e20-4009-98ba-317c8cd063f1" (UID: "3f46fb9c-0e20-4009-98ba-317c8cd063f1"). InnerVolumeSpecName "kube-api-access-69th2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:54:15.248671 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.248614    2587 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f46fb9c-0e20-4009-98ba-317c8cd063f1-console-serving-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:54:15.248671 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.248657    2587 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f46fb9c-0e20-4009-98ba-317c8cd063f1-oauth-serving-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:54:15.248671 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.248666    2587 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f46fb9c-0e20-4009-98ba-317c8cd063f1-console-oauth-config\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:54:15.248671 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.248675    2587 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f46fb9c-0e20-4009-98ba-317c8cd063f1-console-config\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:54:15.248939 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.248684    2587 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f46fb9c-0e20-4009-98ba-317c8cd063f1-service-ca\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:54:15.248939 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.248714    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69th2\" (UniqueName: \"kubernetes.io/projected/3f46fb9c-0e20-4009-98ba-317c8cd063f1-kube-api-access-69th2\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:54:15.248939 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.248724    2587 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f46fb9c-0e20-4009-98ba-317c8cd063f1-trusted-ca-bundle\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:54:15.903145 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.903113    2587 generic.go:356] "Generic (PLEG): container finished" podID="3f46fb9c-0e20-4009-98ba-317c8cd063f1" containerID="9ae0158870dc614c3e2828b522f61718622aff94c76cd5cf6d898968cce65790" exitCode=0
Sep 11 21:54:15.903562 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.903163    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68dbfd879c-8n65p"
Sep 11 21:54:15.903562 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.903202    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68dbfd879c-8n65p" event={"ID":"3f46fb9c-0e20-4009-98ba-317c8cd063f1","Type":"ContainerDied","Data":"9ae0158870dc614c3e2828b522f61718622aff94c76cd5cf6d898968cce65790"}
Sep 11 21:54:15.903562 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.903258    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68dbfd879c-8n65p" event={"ID":"3f46fb9c-0e20-4009-98ba-317c8cd063f1","Type":"ContainerDied","Data":"4b61296a374c073c8a549174d4acebdf0ad3cd8bbf2b0ab7e7af54e7f5ce831f"}
Sep 11 21:54:15.903562 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.903284    2587 scope.go:117] "RemoveContainer" containerID="9ae0158870dc614c3e2828b522f61718622aff94c76cd5cf6d898968cce65790"
Sep 11 21:54:15.911535 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.911515    2587 scope.go:117] "RemoveContainer" containerID="9ae0158870dc614c3e2828b522f61718622aff94c76cd5cf6d898968cce65790"
Sep 11 21:54:15.911805 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:54:15.911784    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ae0158870dc614c3e2828b522f61718622aff94c76cd5cf6d898968cce65790\": container with ID starting with 9ae0158870dc614c3e2828b522f61718622aff94c76cd5cf6d898968cce65790 not found: ID does not exist" containerID="9ae0158870dc614c3e2828b522f61718622aff94c76cd5cf6d898968cce65790"
Sep 11 21:54:15.911869 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.911814    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae0158870dc614c3e2828b522f61718622aff94c76cd5cf6d898968cce65790"} err="failed to get container status \"9ae0158870dc614c3e2828b522f61718622aff94c76cd5cf6d898968cce65790\": rpc error: code = NotFound desc = could not find container \"9ae0158870dc614c3e2828b522f61718622aff94c76cd5cf6d898968cce65790\": container with ID starting with 9ae0158870dc614c3e2828b522f61718622aff94c76cd5cf6d898968cce65790 not found: ID does not exist"
Sep 11 21:54:15.919983 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.919961    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68dbfd879c-8n65p"]
Sep 11 21:54:15.923402 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:15.923383    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68dbfd879c-8n65p"]
Sep 11 21:54:17.334928 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:17.334895    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f46fb9c-0e20-4009-98ba-317c8cd063f1" path="/var/lib/kubelet/pods/3f46fb9c-0e20-4009-98ba-317c8cd063f1/volumes"
Sep 11 21:54:20.810420 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:20.810380    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hxztv"
Sep 11 21:54:22.462566 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:22.462541    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:54:22.466380 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:54:22.466356    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5b7987cc77-2xgzc"
Sep 11 21:55:00.754034 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:00.753992    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74996bdd8f-x8pfn"]
Sep 11 21:55:00.754510 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:00.754276    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f46fb9c-0e20-4009-98ba-317c8cd063f1" containerName="console"
Sep 11 21:55:00.754510 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:00.754288    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f46fb9c-0e20-4009-98ba-317c8cd063f1" containerName="console"
Sep 11 21:55:00.754510 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:00.754357    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f46fb9c-0e20-4009-98ba-317c8cd063f1" containerName="console"
Sep 11 21:55:00.757384 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:00.757365    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:00.760165 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:00.760131    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 21:55:00.760165 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:00.760138    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 21:55:00.760358 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:00.760302    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 21:55:00.760358 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:00.760347    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 21:55:00.760511 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:00.760494    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-q495q\""
Sep 11 21:55:00.760579 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:00.760565    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 21:55:00.765841 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:00.765822    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 21:55:00.771916 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:00.771885    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74996bdd8f-x8pfn"]
Sep 11 21:55:00.905444 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:00.905406    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2f7z\" (UniqueName: \"kubernetes.io/projected/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-kube-api-access-l2f7z\") pod \"console-74996bdd8f-x8pfn\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") " pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:00.905444 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:00.905446    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-trusted-ca-bundle\") pod \"console-74996bdd8f-x8pfn\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") " pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:00.905668 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:00.905468    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-console-oauth-config\") pod \"console-74996bdd8f-x8pfn\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") " pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:00.905668 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:00.905535    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-console-config\") pod \"console-74996bdd8f-x8pfn\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") " pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:00.905668 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:00.905564    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-service-ca\") pod \"console-74996bdd8f-x8pfn\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") " pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:00.905668 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:00.905590    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-oauth-serving-cert\") pod \"console-74996bdd8f-x8pfn\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") " pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:00.905668 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:00.905658    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-console-serving-cert\") pod \"console-74996bdd8f-x8pfn\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") " pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:01.006855 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:01.006786    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-console-serving-cert\") pod \"console-74996bdd8f-x8pfn\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") " pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:01.006855 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:01.006833    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2f7z\" (UniqueName: \"kubernetes.io/projected/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-kube-api-access-l2f7z\") pod \"console-74996bdd8f-x8pfn\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") " pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:01.006855 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:01.006851    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-trusted-ca-bundle\") pod \"console-74996bdd8f-x8pfn\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") " pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:01.007109 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:01.007089    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-console-oauth-config\") pod \"console-74996bdd8f-x8pfn\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") " pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:01.007170 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:01.007147    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-console-config\") pod \"console-74996bdd8f-x8pfn\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") " pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:01.007220 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:01.007176    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-service-ca\") pod \"console-74996bdd8f-x8pfn\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") " pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:01.007271 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:01.007232    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-oauth-serving-cert\") pod \"console-74996bdd8f-x8pfn\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") " pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:01.007707 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:01.007682    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-trusted-ca-bundle\") pod \"console-74996bdd8f-x8pfn\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") " pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:01.007886 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:01.007857    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-service-ca\") pod \"console-74996bdd8f-x8pfn\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") " pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:01.007983 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:01.007891    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-console-config\") pod \"console-74996bdd8f-x8pfn\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") " pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:01.007983 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:01.007867    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-oauth-serving-cert\") pod \"console-74996bdd8f-x8pfn\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") " pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:01.009459 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:01.009442    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-console-oauth-config\") pod \"console-74996bdd8f-x8pfn\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") " pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:01.009508 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:01.009457    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-console-serving-cert\") pod \"console-74996bdd8f-x8pfn\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") " pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:01.014729 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:01.014699    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2f7z\" (UniqueName: \"kubernetes.io/projected/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-kube-api-access-l2f7z\") pod \"console-74996bdd8f-x8pfn\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") " pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:01.066591 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:01.066563    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:01.340703 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:01.340534    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74996bdd8f-x8pfn"]
Sep 11 21:55:01.343581 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:55:01.343555    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod365e94d9_4eba_4b0f_a851_8aaa665e2ed4.slice/crio-f58a439803bd048b4bb4dba7d08db3700c67af4f3371ae3f999d31be50296458 WatchSource:0}: Error finding container f58a439803bd048b4bb4dba7d08db3700c67af4f3371ae3f999d31be50296458: Status 404 returned error can't find the container with id f58a439803bd048b4bb4dba7d08db3700c67af4f3371ae3f999d31be50296458
Sep 11 21:55:02.073302 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:02.073265    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74996bdd8f-x8pfn" event={"ID":"365e94d9-4eba-4b0f-a851-8aaa665e2ed4","Type":"ContainerStarted","Data":"b13334191a1adc1d41b89eee62b014f9b019f84bfee9ce19b44d39ff61391464"}
Sep 11 21:55:02.073687 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:02.073323    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74996bdd8f-x8pfn" event={"ID":"365e94d9-4eba-4b0f-a851-8aaa665e2ed4","Type":"ContainerStarted","Data":"f58a439803bd048b4bb4dba7d08db3700c67af4f3371ae3f999d31be50296458"}
Sep 11 21:55:11.066711 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:11.066669    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:11.067108 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:11.066754    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:11.071867 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:11.071845    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:55:11.090772 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:11.090721    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74996bdd8f-x8pfn" podStartSLOduration=11.090702974 podStartE2EDuration="11.090702974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:55:02.09528116 +0000 UTC m=+155.270357530" watchObservedRunningTime="2026-09-11 21:55:11.090702974 +0000 UTC m=+164.265779347"
Sep 11 21:55:11.109896 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:55:11.109868    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:56:06.836702 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:06.836669    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-pkk6s"]
Sep 11 21:56:06.839864 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:06.839843    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-pkk6s"
Sep 11 21:56:06.842924 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:06.842899    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 21:56:06.844274 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:06.844252    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-lrnvl\""
Sep 11 21:56:06.844411 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:06.844367    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 21:56:06.849243 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:06.849221    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-pkk6s"]
Sep 11 21:56:06.923968 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:06.923936    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvh9c\" (UniqueName: \"kubernetes.io/projected/fd98bb3d-5507-44be-bb69-f694aa768981-kube-api-access-fvh9c\") pod \"cert-manager-webhook-6486ffd997-pkk6s\" (UID: \"fd98bb3d-5507-44be-bb69-f694aa768981\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pkk6s"
Sep 11 21:56:06.924143 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:06.924010    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd98bb3d-5507-44be-bb69-f694aa768981-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-pkk6s\" (UID: \"fd98bb3d-5507-44be-bb69-f694aa768981\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pkk6s"
Sep 11 21:56:07.025255 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:07.025205    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd98bb3d-5507-44be-bb69-f694aa768981-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-pkk6s\" (UID: \"fd98bb3d-5507-44be-bb69-f694aa768981\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pkk6s"
Sep 11 21:56:07.025417 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:07.025272    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvh9c\" (UniqueName: \"kubernetes.io/projected/fd98bb3d-5507-44be-bb69-f694aa768981-kube-api-access-fvh9c\") pod \"cert-manager-webhook-6486ffd997-pkk6s\" (UID: \"fd98bb3d-5507-44be-bb69-f694aa768981\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pkk6s"
Sep 11 21:56:07.033485 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:07.033461    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fd98bb3d-5507-44be-bb69-f694aa768981-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-pkk6s\" (UID: \"fd98bb3d-5507-44be-bb69-f694aa768981\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pkk6s"
Sep 11 21:56:07.033813 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:07.033790    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvh9c\" (UniqueName: \"kubernetes.io/projected/fd98bb3d-5507-44be-bb69-f694aa768981-kube-api-access-fvh9c\") pod \"cert-manager-webhook-6486ffd997-pkk6s\" (UID: \"fd98bb3d-5507-44be-bb69-f694aa768981\") " pod="cert-manager/cert-manager-webhook-6486ffd997-pkk6s"
Sep 11 21:56:07.149177 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:07.149085    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-pkk6s"
Sep 11 21:56:07.417983 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:07.417875    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-pkk6s"]
Sep 11 21:56:07.420635 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:56:07.420605    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd98bb3d_5507_44be_bb69_f694aa768981.slice/crio-9e290df15802363516dc64e5722968237152bbac924b268080b258ad64010e55 WatchSource:0}: Error finding container 9e290df15802363516dc64e5722968237152bbac924b268080b258ad64010e55: Status 404 returned error can't find the container with id 9e290df15802363516dc64e5722968237152bbac924b268080b258ad64010e55
Sep 11 21:56:08.312013 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:08.311980    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-pkk6s" event={"ID":"fd98bb3d-5507-44be-bb69-f694aa768981","Type":"ContainerStarted","Data":"9e290df15802363516dc64e5722968237152bbac924b268080b258ad64010e55"}
Sep 11 21:56:11.325039 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:11.325003    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-pkk6s" event={"ID":"fd98bb3d-5507-44be-bb69-f694aa768981","Type":"ContainerStarted","Data":"a6ec143af08aec1d04343311270a86bbe81f0600ad5f712c6cf83c917665b4db"}
Sep 11 21:56:11.325472 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:11.325057    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-pkk6s"
Sep 11 21:56:11.343817 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:11.343767    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-pkk6s" podStartSLOduration=1.887501037 podStartE2EDuration="5.343752036s" totalImagesPullingTime="3.456250999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:56:11.343247431 +0000 UTC m=+224.518323820" watchObservedRunningTime="2026-09-11 21:56:11.343752036 +0000 UTC m=+224.518828406"
Sep 11 21:56:14.029241 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:14.029204    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-kvkz5"]
Sep 11 21:56:14.032630 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:14.032612    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kvkz5"
Sep 11 21:56:14.035323 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:14.035302    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-zx5hh\""
Sep 11 21:56:14.044322 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:14.044301    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-kvkz5"]
Sep 11 21:56:14.189598 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:14.189556    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szh67\" (UniqueName: \"kubernetes.io/projected/e95c7409-463c-410b-8129-f1cb7b551cd0-kube-api-access-szh67\") pod \"cert-manager-cainjector-559cdd8c87-kvkz5\" (UID: \"e95c7409-463c-410b-8129-f1cb7b551cd0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kvkz5"
Sep 11 21:56:14.189598 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:14.189603    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e95c7409-463c-410b-8129-f1cb7b551cd0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-kvkz5\" (UID: \"e95c7409-463c-410b-8129-f1cb7b551cd0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kvkz5"
Sep 11 21:56:14.291016 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:14.290928    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szh67\" (UniqueName: \"kubernetes.io/projected/e95c7409-463c-410b-8129-f1cb7b551cd0-kube-api-access-szh67\") pod \"cert-manager-cainjector-559cdd8c87-kvkz5\" (UID: \"e95c7409-463c-410b-8129-f1cb7b551cd0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kvkz5"
Sep 11 21:56:14.291016 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:14.290971    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e95c7409-463c-410b-8129-f1cb7b551cd0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-kvkz5\" (UID: \"e95c7409-463c-410b-8129-f1cb7b551cd0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kvkz5"
Sep 11 21:56:14.299639 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:14.299608    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e95c7409-463c-410b-8129-f1cb7b551cd0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-kvkz5\" (UID: \"e95c7409-463c-410b-8129-f1cb7b551cd0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kvkz5"
Sep 11 21:56:14.299836 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:14.299812    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szh67\" (UniqueName: \"kubernetes.io/projected/e95c7409-463c-410b-8129-f1cb7b551cd0-kube-api-access-szh67\") pod \"cert-manager-cainjector-559cdd8c87-kvkz5\" (UID: \"e95c7409-463c-410b-8129-f1cb7b551cd0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kvkz5"
Sep 11 21:56:14.340971 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:14.340944    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kvkz5"
Sep 11 21:56:14.612096 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:14.612024    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-kvkz5"]
Sep 11 21:56:14.615088 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:56:14.615062    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode95c7409_463c_410b_8129_f1cb7b551cd0.slice/crio-eb0a0f8412fad51fbcf8b7740e8bc482416c9128266bc4b82d7d778509bbf779 WatchSource:0}: Error finding container eb0a0f8412fad51fbcf8b7740e8bc482416c9128266bc4b82d7d778509bbf779: Status 404 returned error can't find the container with id eb0a0f8412fad51fbcf8b7740e8bc482416c9128266bc4b82d7d778509bbf779
Sep 11 21:56:15.338738 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:15.338701    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kvkz5" event={"ID":"e95c7409-463c-410b-8129-f1cb7b551cd0","Type":"ContainerStarted","Data":"c247b40b649c299fa6f007464876e78781bcdf85bcca230a94cd5c071ea78cfa"}
Sep 11 21:56:15.338738 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:15.338743    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kvkz5" event={"ID":"e95c7409-463c-410b-8129-f1cb7b551cd0","Type":"ContainerStarted","Data":"eb0a0f8412fad51fbcf8b7740e8bc482416c9128266bc4b82d7d778509bbf779"}
Sep 11 21:56:15.355401 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:15.355344    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kvkz5" podStartSLOduration=1.355309767 podStartE2EDuration="1.355309767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:56:15.354123092 +0000 UTC m=+228.529199490" watchObservedRunningTime="2026-09-11 21:56:15.355309767 +0000 UTC m=+228.530386137"
Sep 11 21:56:17.334182 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:17.334153    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-pkk6s"
Sep 11 21:56:43.704790 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:43.704754    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6"]
Sep 11 21:56:43.748267 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:43.748222    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6"]
Sep 11 21:56:43.748440 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:43.748366    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6"
Sep 11 21:56:43.751251 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:43.751222    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 21:56:43.751465 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:43.751345    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 21:56:43.751626 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:43.751612    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-bfn6m\""
Sep 11 21:56:43.751801 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:43.751785    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 21:56:43.753643 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:43.753624    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 21:56:43.810290 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:43.810250    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vsd6\" (UniqueName: \"kubernetes.io/projected/b88b394c-61ba-4af9-b3d3-fb5ef027cefe-kube-api-access-6vsd6\") pod \"opendatahub-operator-controller-manager-5bf8845876-7pms6\" (UID: \"b88b394c-61ba-4af9-b3d3-fb5ef027cefe\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6"
Sep 11 21:56:43.810503 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:43.810307    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b88b394c-61ba-4af9-b3d3-fb5ef027cefe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bf8845876-7pms6\" (UID: \"b88b394c-61ba-4af9-b3d3-fb5ef027cefe\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6"
Sep 11 21:56:43.810503 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:43.810387    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b88b394c-61ba-4af9-b3d3-fb5ef027cefe-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bf8845876-7pms6\" (UID: \"b88b394c-61ba-4af9-b3d3-fb5ef027cefe\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6"
Sep 11 21:56:43.810503 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:43.810416    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b88b394c-61ba-4af9-b3d3-fb5ef027cefe-manifests\") pod \"opendatahub-operator-controller-manager-5bf8845876-7pms6\" (UID: \"b88b394c-61ba-4af9-b3d3-fb5ef027cefe\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6"
Sep 11 21:56:43.810503 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:43.810442    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b88b394c-61ba-4af9-b3d3-fb5ef027cefe-tmp\") pod \"opendatahub-operator-controller-manager-5bf8845876-7pms6\" (UID: \"b88b394c-61ba-4af9-b3d3-fb5ef027cefe\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6"
Sep 11 21:56:43.911506 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:43.911475    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b88b394c-61ba-4af9-b3d3-fb5ef027cefe-tmp\") pod \"opendatahub-operator-controller-manager-5bf8845876-7pms6\" (UID: \"b88b394c-61ba-4af9-b3d3-fb5ef027cefe\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6"
Sep 11 21:56:43.911691 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:43.911535    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vsd6\" (UniqueName: \"kubernetes.io/projected/b88b394c-61ba-4af9-b3d3-fb5ef027cefe-kube-api-access-6vsd6\") pod \"opendatahub-operator-controller-manager-5bf8845876-7pms6\" (UID: \"b88b394c-61ba-4af9-b3d3-fb5ef027cefe\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6"
Sep 11 21:56:43.911691 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:43.911589    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b88b394c-61ba-4af9-b3d3-fb5ef027cefe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bf8845876-7pms6\" (UID: \"b88b394c-61ba-4af9-b3d3-fb5ef027cefe\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6"
Sep 11 21:56:43.911691 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:43.911610    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b88b394c-61ba-4af9-b3d3-fb5ef027cefe-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bf8845876-7pms6\" (UID: \"b88b394c-61ba-4af9-b3d3-fb5ef027cefe\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6"
Sep 11 21:56:43.911691 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:43.911625    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b88b394c-61ba-4af9-b3d3-fb5ef027cefe-manifests\") pod \"opendatahub-operator-controller-manager-5bf8845876-7pms6\" (UID: \"b88b394c-61ba-4af9-b3d3-fb5ef027cefe\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6"
Sep 11 21:56:43.912045 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:43.912012    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b88b394c-61ba-4af9-b3d3-fb5ef027cefe-manifests\") pod \"opendatahub-operator-controller-manager-5bf8845876-7pms6\" (UID: \"b88b394c-61ba-4af9-b3d3-fb5ef027cefe\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6"
Sep 11 21:56:43.914004 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:43.913978    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b88b394c-61ba-4af9-b3d3-fb5ef027cefe-tmp\") pod \"opendatahub-operator-controller-manager-5bf8845876-7pms6\" (UID: \"b88b394c-61ba-4af9-b3d3-fb5ef027cefe\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6"
Sep 11 21:56:43.914152 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:43.914133    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b88b394c-61ba-4af9-b3d3-fb5ef027cefe-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5bf8845876-7pms6\" (UID: \"b88b394c-61ba-4af9-b3d3-fb5ef027cefe\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6"
Sep 11 21:56:43.914195 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:43.914152    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b88b394c-61ba-4af9-b3d3-fb5ef027cefe-webhook-cert\") pod \"opendatahub-operator-controller-manager-5bf8845876-7pms6\" (UID: \"b88b394c-61ba-4af9-b3d3-fb5ef027cefe\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6"
Sep 11 21:56:43.925239 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:43.925215    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vsd6\" (UniqueName: \"kubernetes.io/projected/b88b394c-61ba-4af9-b3d3-fb5ef027cefe-kube-api-access-6vsd6\") pod \"opendatahub-operator-controller-manager-5bf8845876-7pms6\" (UID: \"b88b394c-61ba-4af9-b3d3-fb5ef027cefe\") " pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6"
Sep 11 21:56:44.059214 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:44.059115    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6"
Sep 11 21:56:44.367074 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:44.367047    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6"]
Sep 11 21:56:44.368413 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:56:44.368384    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88b394c_61ba_4af9_b3d3_fb5ef027cefe.slice/crio-c47331d601e79e9153be9d1894c8ee8de4521543c003b9e073ffc8e22ef5fb44 WatchSource:0}: Error finding container c47331d601e79e9153be9d1894c8ee8de4521543c003b9e073ffc8e22ef5fb44: Status 404 returned error can't find the container with id c47331d601e79e9153be9d1894c8ee8de4521543c003b9e073ffc8e22ef5fb44
Sep 11 21:56:44.446191 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:44.446150    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6" event={"ID":"b88b394c-61ba-4af9-b3d3-fb5ef027cefe","Type":"ContainerStarted","Data":"c47331d601e79e9153be9d1894c8ee8de4521543c003b9e073ffc8e22ef5fb44"}
Sep 11 21:56:47.462600 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:47.462558    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6" event={"ID":"b88b394c-61ba-4af9-b3d3-fb5ef027cefe","Type":"ContainerStarted","Data":"e4d0c038e5307a281efea6f6c108d4257d5c8cf13656bf88b513c21639600446"}
Sep 11 21:56:48.468599 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:48.468564    2587 generic.go:356] "Generic (PLEG): container finished" podID="b88b394c-61ba-4af9-b3d3-fb5ef027cefe" containerID="e4d0c038e5307a281efea6f6c108d4257d5c8cf13656bf88b513c21639600446" exitCode=0
Sep 11 21:56:48.468967 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:48.468635    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6" event={"ID":"b88b394c-61ba-4af9-b3d3-fb5ef027cefe","Type":"ContainerDied","Data":"e4d0c038e5307a281efea6f6c108d4257d5c8cf13656bf88b513c21639600446"}
Sep 11 21:56:49.474508 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:49.474470    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6" event={"ID":"b88b394c-61ba-4af9-b3d3-fb5ef027cefe","Type":"ContainerStarted","Data":"b485e4059e5f04e39bae857317c02902e7ebd5becd3200ebf40a22e66f793d67"}
Sep 11 21:56:49.474900 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:49.474558    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6"
Sep 11 21:56:49.504042 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:56:49.503987    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6" podStartSLOduration=2.132428519 podStartE2EDuration="6.503969666s" totalImagesPullingTime="3.350331735s" totalInitContainerRuntime="1.021209412s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:56:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:56:49.502288289 +0000 UTC m=+262.677364676" watchObservedRunningTime="2026-09-11 21:56:49.503969666 +0000 UTC m=+262.679046036"
Sep 11 21:57:00.485288 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:00.485256    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5bf8845876-7pms6"
Sep 11 21:57:04.807656 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:04.807617    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-576b4b4bb7-v29kw"]
Sep 11 21:57:04.816344 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:04.816307    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-v29kw"
Sep 11 21:57:04.819776 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:04.819751    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 21:57:04.820857 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:04.820825    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 21:57:04.820965 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:04.820881    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 21:57:04.820965 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:04.820936    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 21:57:04.821947 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:04.821929    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-lqhj4\""
Sep 11 21:57:04.828289 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:04.828260    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-576b4b4bb7-v29kw"]
Sep 11 21:57:04.893080 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:04.893042    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzxsr\" (UniqueName: \"kubernetes.io/projected/a979a0c2-3065-4255-bae3-458f48999604-kube-api-access-tzxsr\") pod \"kube-auth-proxy-576b4b4bb7-v29kw\" (UID: \"a979a0c2-3065-4255-bae3-458f48999604\") " pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-v29kw"
Sep 11 21:57:04.893262 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:04.893136    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a979a0c2-3065-4255-bae3-458f48999604-tls-certs\") pod \"kube-auth-proxy-576b4b4bb7-v29kw\" (UID: \"a979a0c2-3065-4255-bae3-458f48999604\") " pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-v29kw"
Sep 11 21:57:04.893262 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:04.893176    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a979a0c2-3065-4255-bae3-458f48999604-tmp\") pod \"kube-auth-proxy-576b4b4bb7-v29kw\" (UID: \"a979a0c2-3065-4255-bae3-458f48999604\") " pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-v29kw"
Sep 11 21:57:04.994371 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:04.994309    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a979a0c2-3065-4255-bae3-458f48999604-tls-certs\") pod \"kube-auth-proxy-576b4b4bb7-v29kw\" (UID: \"a979a0c2-3065-4255-bae3-458f48999604\") " pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-v29kw"
Sep 11 21:57:04.994593 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:04.994395    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a979a0c2-3065-4255-bae3-458f48999604-tmp\") pod \"kube-auth-proxy-576b4b4bb7-v29kw\" (UID: \"a979a0c2-3065-4255-bae3-458f48999604\") " pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-v29kw"
Sep 11 21:57:04.994593 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:04.994455    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzxsr\" (UniqueName: \"kubernetes.io/projected/a979a0c2-3065-4255-bae3-458f48999604-kube-api-access-tzxsr\") pod \"kube-auth-proxy-576b4b4bb7-v29kw\" (UID: \"a979a0c2-3065-4255-bae3-458f48999604\") " pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-v29kw"
Sep 11 21:57:04.994593 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:57:04.994461    2587 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 11 21:57:04.994593 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:57:04.994538    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a979a0c2-3065-4255-bae3-458f48999604-tls-certs podName:a979a0c2-3065-4255-bae3-458f48999604 nodeName:}" failed. No retries permitted until 2026-09-11 21:57:05.49451932 +0000 UTC m=+278.669595668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a979a0c2-3065-4255-bae3-458f48999604-tls-certs") pod "kube-auth-proxy-576b4b4bb7-v29kw" (UID: "a979a0c2-3065-4255-bae3-458f48999604") : secret "kube-auth-proxy-tls" not found
Sep 11 21:57:04.996741 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:04.996715    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a979a0c2-3065-4255-bae3-458f48999604-tmp\") pod \"kube-auth-proxy-576b4b4bb7-v29kw\" (UID: \"a979a0c2-3065-4255-bae3-458f48999604\") " pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-v29kw"
Sep 11 21:57:05.010435 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:05.010401    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzxsr\" (UniqueName: \"kubernetes.io/projected/a979a0c2-3065-4255-bae3-458f48999604-kube-api-access-tzxsr\") pod \"kube-auth-proxy-576b4b4bb7-v29kw\" (UID: \"a979a0c2-3065-4255-bae3-458f48999604\") " pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-v29kw"
Sep 11 21:57:05.500462 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:05.500419    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a979a0c2-3065-4255-bae3-458f48999604-tls-certs\") pod \"kube-auth-proxy-576b4b4bb7-v29kw\" (UID: \"a979a0c2-3065-4255-bae3-458f48999604\") " pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-v29kw"
Sep 11 21:57:05.502979 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:05.502959    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a979a0c2-3065-4255-bae3-458f48999604-tls-certs\") pod \"kube-auth-proxy-576b4b4bb7-v29kw\" (UID: \"a979a0c2-3065-4255-bae3-458f48999604\") " pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-v29kw"
Sep 11 21:57:05.726437 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:05.726396    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-v29kw"
Sep 11 21:57:06.017110 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.017072    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-576b4b4bb7-v29kw"]
Sep 11 21:57:06.020456 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:57:06.020426    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda979a0c2_3065_4255_bae3_458f48999604.slice/crio-f675692bc8c1eaa0bf5e5699aa5a7f5d31c77aa592450a385fc19b05cbf83055 WatchSource:0}: Error finding container f675692bc8c1eaa0bf5e5699aa5a7f5d31c77aa592450a385fc19b05cbf83055: Status 404 returned error can't find the container with id f675692bc8c1eaa0bf5e5699aa5a7f5d31c77aa592450a385fc19b05cbf83055
Sep 11 21:57:06.425807 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.425773    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"]
Sep 11 21:57:06.430206 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.430190    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:06.433279 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.433249    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 11 21:57:06.434066 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.434025    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 11 21:57:06.434193 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.434035    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xmxph\""
Sep 11 21:57:06.434193 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.434163    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 21:57:06.434835 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.434817    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 11 21:57:06.450765 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.450736    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"]
Sep 11 21:57:06.510088 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.510050    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/07d3dee0-485f-4229-b5c2-5994feb13679-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bfxb7\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:06.510239 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.510097    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pbj2\" (UniqueName: \"kubernetes.io/projected/07d3dee0-485f-4229-b5c2-5994feb13679-kube-api-access-2pbj2\") pod \"istiod-openshift-gateway-7b7995d5cd-bfxb7\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:06.510239 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.510178    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07d3dee0-485f-4229-b5c2-5994feb13679-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bfxb7\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:06.510239 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.510221    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/07d3dee0-485f-4229-b5c2-5994feb13679-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bfxb7\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:06.510394 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.510242    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/07d3dee0-485f-4229-b5c2-5994feb13679-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bfxb7\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:06.510394 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.510283    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/07d3dee0-485f-4229-b5c2-5994feb13679-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bfxb7\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:06.510394 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.510303    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/07d3dee0-485f-4229-b5c2-5994feb13679-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bfxb7\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:06.552930 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.552891    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-v29kw" event={"ID":"a979a0c2-3065-4255-bae3-458f48999604","Type":"ContainerStarted","Data":"f675692bc8c1eaa0bf5e5699aa5a7f5d31c77aa592450a385fc19b05cbf83055"}
Sep 11 21:57:06.611217 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.611168    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/07d3dee0-485f-4229-b5c2-5994feb13679-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bfxb7\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:06.611445 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.611227    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/07d3dee0-485f-4229-b5c2-5994feb13679-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bfxb7\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:06.611445 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.611270    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/07d3dee0-485f-4229-b5c2-5994feb13679-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bfxb7\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:06.611445 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.611316    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pbj2\" (UniqueName: \"kubernetes.io/projected/07d3dee0-485f-4229-b5c2-5994feb13679-kube-api-access-2pbj2\") pod \"istiod-openshift-gateway-7b7995d5cd-bfxb7\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:06.611445 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.611397    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07d3dee0-485f-4229-b5c2-5994feb13679-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bfxb7\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:06.611682 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.611484    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/07d3dee0-485f-4229-b5c2-5994feb13679-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bfxb7\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:06.611682 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.611543    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/07d3dee0-485f-4229-b5c2-5994feb13679-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bfxb7\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:06.612256 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.612189    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/07d3dee0-485f-4229-b5c2-5994feb13679-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bfxb7\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:06.614497 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.614446    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/07d3dee0-485f-4229-b5c2-5994feb13679-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bfxb7\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:06.614744 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.614700    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/07d3dee0-485f-4229-b5c2-5994feb13679-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bfxb7\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:06.615032 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.615012    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/07d3dee0-485f-4229-b5c2-5994feb13679-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bfxb7\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:06.615872 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.615827    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/07d3dee0-485f-4229-b5c2-5994feb13679-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bfxb7\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:06.620904 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.620875    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07d3dee0-485f-4229-b5c2-5994feb13679-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bfxb7\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:06.621862 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.621818    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pbj2\" (UniqueName: \"kubernetes.io/projected/07d3dee0-485f-4229-b5c2-5994feb13679-kube-api-access-2pbj2\") pod \"istiod-openshift-gateway-7b7995d5cd-bfxb7\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:06.741472 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:06.741340    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:07.045425 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:07.045387    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"]
Sep 11 21:57:07.048195 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:57:07.048157    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d3dee0_485f_4229_b5c2_5994feb13679.slice/crio-c5827910d0ba2b8f2c4531be5354701d9062a48b680057dc11508e7ba435c773 WatchSource:0}: Error finding container c5827910d0ba2b8f2c4531be5354701d9062a48b680057dc11508e7ba435c773: Status 404 returned error can't find the container with id c5827910d0ba2b8f2c4531be5354701d9062a48b680057dc11508e7ba435c773
Sep 11 21:57:07.559923 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:07.559882    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7" event={"ID":"07d3dee0-485f-4229-b5c2-5994feb13679","Type":"ContainerStarted","Data":"c5827910d0ba2b8f2c4531be5354701d9062a48b680057dc11508e7ba435c773"}
Sep 11 21:57:09.571768 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:09.571707    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-v29kw" event={"ID":"a979a0c2-3065-4255-bae3-458f48999604","Type":"ContainerStarted","Data":"ae4c89ecbe445507913dd89ab1c6850b36701b944e74c46a95d9ceaec7ce7dfa"}
Sep 11 21:57:09.589994 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:09.589936    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-576b4b4bb7-v29kw" podStartSLOduration=2.792895811 podStartE2EDuration="5.589916234s" totalImagesPullingTime="2.797020423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:57:09.588459511 +0000 UTC m=+282.763535899" watchObservedRunningTime="2026-09-11 21:57:09.589916234 +0000 UTC m=+282.764992630"
Sep 11 21:57:10.587210 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:10.587174    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 21:57:10.587487 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:10.587248    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 21:57:11.582769 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:11.582731    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7" event={"ID":"07d3dee0-485f-4229-b5c2-5994feb13679","Type":"ContainerStarted","Data":"75ec1acf1bf9d4c52dbcd40d68de1b8d1be285fd6d4bd6f388003cc5c35cb6a4"}
Sep 11 21:57:11.584486 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:11.584458    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:11.585205 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:11.585176    2587 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-bfxb7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 21:57:11.585306 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:11.585226    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7" podUID="07d3dee0-485f-4229-b5c2-5994feb13679" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 21:57:11.610599 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:11.610537    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7" podStartSLOduration=2.075080145 podStartE2EDuration="5.61051369s" totalImagesPullingTime="3.535433545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:57:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:57:11.607976636 +0000 UTC m=+284.783053033" watchObservedRunningTime="2026-09-11 21:57:11.61051369 +0000 UTC m=+284.785590061"
Sep 11 21:57:12.589065 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:12.589038    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:25.615781 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:25.615742    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-rlw6w"]
Sep 11 21:57:25.625187 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:25.625161    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rlw6w"
Sep 11 21:57:25.628592 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:25.628543    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 11 21:57:25.628592 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:25.628582    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-txfxd\""
Sep 11 21:57:25.634914 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:25.634889    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lkxdk"]
Sep 11 21:57:25.638797 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:25.638581    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lkxdk"
Sep 11 21:57:25.641316 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:25.641017    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-rlw6w"]
Sep 11 21:57:25.642321 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:25.642274    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 11 21:57:25.644066 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:25.644046    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-d9qrq\""
Sep 11 21:57:25.656095 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:25.656059    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lkxdk"]
Sep 11 21:57:25.670350 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:25.670294    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxq8t\" (UniqueName: \"kubernetes.io/projected/c6895de2-3e55-4c67-8107-7ed453c7a934-kube-api-access-bxq8t\") pod \"kserve-controller-manager-d6f4b65b8-rlw6w\" (UID: \"c6895de2-3e55-4c67-8107-7ed453c7a934\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rlw6w"
Sep 11 21:57:25.670529 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:25.670385    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcwxv\" (UniqueName: \"kubernetes.io/projected/c57c2e4e-3ab4-4276-9016-b9ff4a0202fc-kube-api-access-gcwxv\") pod \"llmisvc-controller-manager-777c6bd7d5-lkxdk\" (UID: \"c57c2e4e-3ab4-4276-9016-b9ff4a0202fc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lkxdk"
Sep 11 21:57:25.670529 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:25.670434    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c57c2e4e-3ab4-4276-9016-b9ff4a0202fc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lkxdk\" (UID: \"c57c2e4e-3ab4-4276-9016-b9ff4a0202fc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lkxdk"
Sep 11 21:57:25.670529 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:25.670495    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6895de2-3e55-4c67-8107-7ed453c7a934-cert\") pod \"kserve-controller-manager-d6f4b65b8-rlw6w\" (UID: \"c6895de2-3e55-4c67-8107-7ed453c7a934\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rlw6w"
Sep 11 21:57:25.770958 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:25.770914    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxq8t\" (UniqueName: \"kubernetes.io/projected/c6895de2-3e55-4c67-8107-7ed453c7a934-kube-api-access-bxq8t\") pod \"kserve-controller-manager-d6f4b65b8-rlw6w\" (UID: \"c6895de2-3e55-4c67-8107-7ed453c7a934\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rlw6w"
Sep 11 21:57:25.771161 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:25.770970    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcwxv\" (UniqueName: \"kubernetes.io/projected/c57c2e4e-3ab4-4276-9016-b9ff4a0202fc-kube-api-access-gcwxv\") pod \"llmisvc-controller-manager-777c6bd7d5-lkxdk\" (UID: \"c57c2e4e-3ab4-4276-9016-b9ff4a0202fc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lkxdk"
Sep 11 21:57:25.771161 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:25.771023    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c57c2e4e-3ab4-4276-9016-b9ff4a0202fc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lkxdk\" (UID: \"c57c2e4e-3ab4-4276-9016-b9ff4a0202fc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lkxdk"
Sep 11 21:57:25.771161 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:25.771076    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6895de2-3e55-4c67-8107-7ed453c7a934-cert\") pod \"kserve-controller-manager-d6f4b65b8-rlw6w\" (UID: \"c6895de2-3e55-4c67-8107-7ed453c7a934\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rlw6w"
Sep 11 21:57:25.771357 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:57:25.771178    2587 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 11 21:57:25.771357 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:57:25.771206    2587 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 11 21:57:25.771357 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:57:25.771260    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c57c2e4e-3ab4-4276-9016-b9ff4a0202fc-cert podName:c57c2e4e-3ab4-4276-9016-b9ff4a0202fc nodeName:}" failed. No retries permitted until 2026-09-11 21:57:26.271236818 +0000 UTC m=+299.446313181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c57c2e4e-3ab4-4276-9016-b9ff4a0202fc-cert") pod "llmisvc-controller-manager-777c6bd7d5-lkxdk" (UID: "c57c2e4e-3ab4-4276-9016-b9ff4a0202fc") : secret "llmisvc-webhook-server-cert" not found
Sep 11 21:57:25.771357 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:57:25.771279    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6895de2-3e55-4c67-8107-7ed453c7a934-cert podName:c6895de2-3e55-4c67-8107-7ed453c7a934 nodeName:}" failed. No retries permitted until 2026-09-11 21:57:26.271270227 +0000 UTC m=+299.446346579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/c6895de2-3e55-4c67-8107-7ed453c7a934-cert") pod "kserve-controller-manager-d6f4b65b8-rlw6w" (UID: "c6895de2-3e55-4c67-8107-7ed453c7a934") : secret "kserve-webhook-server-cert" not found
Sep 11 21:57:25.780271 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:25.780245    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxq8t\" (UniqueName: \"kubernetes.io/projected/c6895de2-3e55-4c67-8107-7ed453c7a934-kube-api-access-bxq8t\") pod \"kserve-controller-manager-d6f4b65b8-rlw6w\" (UID: \"c6895de2-3e55-4c67-8107-7ed453c7a934\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rlw6w"
Sep 11 21:57:25.788287 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:25.788257    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcwxv\" (UniqueName: \"kubernetes.io/projected/c57c2e4e-3ab4-4276-9016-b9ff4a0202fc-kube-api-access-gcwxv\") pod \"llmisvc-controller-manager-777c6bd7d5-lkxdk\" (UID: \"c57c2e4e-3ab4-4276-9016-b9ff4a0202fc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lkxdk"
Sep 11 21:57:26.275682 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:26.275640    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c57c2e4e-3ab4-4276-9016-b9ff4a0202fc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lkxdk\" (UID: \"c57c2e4e-3ab4-4276-9016-b9ff4a0202fc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lkxdk"
Sep 11 21:57:26.275866 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:26.275698    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6895de2-3e55-4c67-8107-7ed453c7a934-cert\") pod \"kserve-controller-manager-d6f4b65b8-rlw6w\" (UID: \"c6895de2-3e55-4c67-8107-7ed453c7a934\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rlw6w"
Sep 11 21:57:26.278136 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:26.278110    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6895de2-3e55-4c67-8107-7ed453c7a934-cert\") pod \"kserve-controller-manager-d6f4b65b8-rlw6w\" (UID: \"c6895de2-3e55-4c67-8107-7ed453c7a934\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rlw6w"
Sep 11 21:57:26.278244 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:26.278185    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c57c2e4e-3ab4-4276-9016-b9ff4a0202fc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-lkxdk\" (UID: \"c57c2e4e-3ab4-4276-9016-b9ff4a0202fc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lkxdk"
Sep 11 21:57:26.536381 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:26.536279    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rlw6w"
Sep 11 21:57:26.550708 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:26.550485    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lkxdk"
Sep 11 21:57:26.830045 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:26.830018    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-rlw6w"]
Sep 11 21:57:26.831779 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:57:26.831751    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6895de2_3e55_4c67_8107_7ed453c7a934.slice/crio-b7b180953bb3f1e21c0b0fb217f7e6398b9ce2f3d95b9c1341cb4348348418c2 WatchSource:0}: Error finding container b7b180953bb3f1e21c0b0fb217f7e6398b9ce2f3d95b9c1341cb4348348418c2: Status 404 returned error can't find the container with id b7b180953bb3f1e21c0b0fb217f7e6398b9ce2f3d95b9c1341cb4348348418c2
Sep 11 21:57:26.851803 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:26.851782    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-lkxdk"]
Sep 11 21:57:26.853607 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:57:26.853583    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc57c2e4e_3ab4_4276_9016_b9ff4a0202fc.slice/crio-a29885d93d7a596163ccbfb2a1d932db9938e7cc3c3d9632ff44bfaac262d910 WatchSource:0}: Error finding container a29885d93d7a596163ccbfb2a1d932db9938e7cc3c3d9632ff44bfaac262d910: Status 404 returned error can't find the container with id a29885d93d7a596163ccbfb2a1d932db9938e7cc3c3d9632ff44bfaac262d910
Sep 11 21:57:27.277725 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:27.277121    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw6qf_7ab8a820-c9ec-415c-bfaa-3c9744ac040a/ovn-acl-logging/0.log"
Sep 11 21:57:27.281069 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:27.281046    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw6qf_7ab8a820-c9ec-415c-bfaa-3c9744ac040a/ovn-acl-logging/0.log"
Sep 11 21:57:27.281480 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:27.281461    2587 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 21:57:27.582811 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:27.582737    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"]
Sep 11 21:57:27.585001 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:27.582986    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7" podUID="07d3dee0-485f-4229-b5c2-5994feb13679" containerName="discovery" containerID="cri-o://75ec1acf1bf9d4c52dbcd40d68de1b8d1be285fd6d4bd6f388003cc5c35cb6a4" gracePeriod=30
Sep 11 21:57:27.589346 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:27.589304    2587 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-bfxb7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.24:8080/ready\": dial tcp 10.134.0.24:8080: connect: connection refused" start-of-body=
Sep 11 21:57:27.589484 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:27.589386    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7" podUID="07d3dee0-485f-4229-b5c2-5994feb13679" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.24:8080/ready\": dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 11 21:57:27.656521 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:27.656483    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lkxdk" event={"ID":"c57c2e4e-3ab4-4276-9016-b9ff4a0202fc","Type":"ContainerStarted","Data":"a29885d93d7a596163ccbfb2a1d932db9938e7cc3c3d9632ff44bfaac262d910"}
Sep 11 21:57:27.658063 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:27.658031    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rlw6w" event={"ID":"c6895de2-3e55-4c67-8107-7ed453c7a934","Type":"ContainerStarted","Data":"b7b180953bb3f1e21c0b0fb217f7e6398b9ce2f3d95b9c1341cb4348348418c2"}
Sep 11 21:57:28.008555 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.008523    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:28.092830 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.092288    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/07d3dee0-485f-4229-b5c2-5994feb13679-local-certs\") pod \"07d3dee0-485f-4229-b5c2-5994feb13679\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") "
Sep 11 21:57:28.092830 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.092359    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/07d3dee0-485f-4229-b5c2-5994feb13679-istio-kubeconfig\") pod \"07d3dee0-485f-4229-b5c2-5994feb13679\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") "
Sep 11 21:57:28.092830 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.092388    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/07d3dee0-485f-4229-b5c2-5994feb13679-istio-csr-ca-configmap\") pod \"07d3dee0-485f-4229-b5c2-5994feb13679\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") "
Sep 11 21:57:28.092830 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.092424    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07d3dee0-485f-4229-b5c2-5994feb13679-istio-token\") pod \"07d3dee0-485f-4229-b5c2-5994feb13679\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") "
Sep 11 21:57:28.092830 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.092472    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/07d3dee0-485f-4229-b5c2-5994feb13679-cacerts\") pod \"07d3dee0-485f-4229-b5c2-5994feb13679\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") "
Sep 11 21:57:28.092830 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.092520    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pbj2\" (UniqueName: \"kubernetes.io/projected/07d3dee0-485f-4229-b5c2-5994feb13679-kube-api-access-2pbj2\") pod \"07d3dee0-485f-4229-b5c2-5994feb13679\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") "
Sep 11 21:57:28.092830 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.092564    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/07d3dee0-485f-4229-b5c2-5994feb13679-istio-csr-dns-cert\") pod \"07d3dee0-485f-4229-b5c2-5994feb13679\" (UID: \"07d3dee0-485f-4229-b5c2-5994feb13679\") "
Sep 11 21:57:28.094653 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.094598    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d3dee0-485f-4229-b5c2-5994feb13679-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "07d3dee0-485f-4229-b5c2-5994feb13679" (UID: "07d3dee0-485f-4229-b5c2-5994feb13679"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:57:28.096667 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.096638    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d3dee0-485f-4229-b5c2-5994feb13679-local-certs" (OuterVolumeSpecName: "local-certs") pod "07d3dee0-485f-4229-b5c2-5994feb13679" (UID: "07d3dee0-485f-4229-b5c2-5994feb13679"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:57:28.099482 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.097522    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d3dee0-485f-4229-b5c2-5994feb13679-istio-token" (OuterVolumeSpecName: "istio-token") pod "07d3dee0-485f-4229-b5c2-5994feb13679" (UID: "07d3dee0-485f-4229-b5c2-5994feb13679"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:57:28.105769 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.101100    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07d3dee0-485f-4229-b5c2-5994feb13679-cacerts" (OuterVolumeSpecName: "cacerts") pod "07d3dee0-485f-4229-b5c2-5994feb13679" (UID: "07d3dee0-485f-4229-b5c2-5994feb13679"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:57:28.105769 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.104483    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07d3dee0-485f-4229-b5c2-5994feb13679-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "07d3dee0-485f-4229-b5c2-5994feb13679" (UID: "07d3dee0-485f-4229-b5c2-5994feb13679"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:57:28.105769 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.105685    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d3dee0-485f-4229-b5c2-5994feb13679-kube-api-access-2pbj2" (OuterVolumeSpecName: "kube-api-access-2pbj2") pod "07d3dee0-485f-4229-b5c2-5994feb13679" (UID: "07d3dee0-485f-4229-b5c2-5994feb13679"). InnerVolumeSpecName "kube-api-access-2pbj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:57:28.106289 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.106181    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07d3dee0-485f-4229-b5c2-5994feb13679-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "07d3dee0-485f-4229-b5c2-5994feb13679" (UID: "07d3dee0-485f-4229-b5c2-5994feb13679"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:57:28.194117 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.194082    2587 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/07d3dee0-485f-4229-b5c2-5994feb13679-local-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:57:28.194117 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.194117    2587 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/07d3dee0-485f-4229-b5c2-5994feb13679-istio-kubeconfig\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:57:28.194367 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.194133    2587 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/07d3dee0-485f-4229-b5c2-5994feb13679-istio-csr-ca-configmap\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:57:28.194367 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.194149    2587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07d3dee0-485f-4229-b5c2-5994feb13679-istio-token\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:57:28.194367 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.194165    2587 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/07d3dee0-485f-4229-b5c2-5994feb13679-cacerts\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:57:28.194367 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.194177    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pbj2\" (UniqueName: \"kubernetes.io/projected/07d3dee0-485f-4229-b5c2-5994feb13679-kube-api-access-2pbj2\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:57:28.194367 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.194190    2587 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/07d3dee0-485f-4229-b5c2-5994feb13679-istio-csr-dns-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:57:28.665681 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.665641    2587 generic.go:356] "Generic (PLEG): container finished" podID="07d3dee0-485f-4229-b5c2-5994feb13679" containerID="75ec1acf1bf9d4c52dbcd40d68de1b8d1be285fd6d4bd6f388003cc5c35cb6a4" exitCode=0
Sep 11 21:57:28.665859 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.665704    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"
Sep 11 21:57:28.665859 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.665732    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7" event={"ID":"07d3dee0-485f-4229-b5c2-5994feb13679","Type":"ContainerDied","Data":"75ec1acf1bf9d4c52dbcd40d68de1b8d1be285fd6d4bd6f388003cc5c35cb6a4"}
Sep 11 21:57:28.665859 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.665795    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7" event={"ID":"07d3dee0-485f-4229-b5c2-5994feb13679","Type":"ContainerDied","Data":"c5827910d0ba2b8f2c4531be5354701d9062a48b680057dc11508e7ba435c773"}
Sep 11 21:57:28.665859 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.665823    2587 scope.go:117] "RemoveContainer" containerID="75ec1acf1bf9d4c52dbcd40d68de1b8d1be285fd6d4bd6f388003cc5c35cb6a4"
Sep 11 21:57:28.700258 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.700228    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"]
Sep 11 21:57:28.713369 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:28.713322    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bfxb7"]
Sep 11 21:57:29.049275 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:29.049252    2587 scope.go:117] "RemoveContainer" containerID="75ec1acf1bf9d4c52dbcd40d68de1b8d1be285fd6d4bd6f388003cc5c35cb6a4"
Sep 11 21:57:29.049697 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:57:29.049669    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ec1acf1bf9d4c52dbcd40d68de1b8d1be285fd6d4bd6f388003cc5c35cb6a4\": container with ID starting with 75ec1acf1bf9d4c52dbcd40d68de1b8d1be285fd6d4bd6f388003cc5c35cb6a4 not found: ID does not exist" containerID="75ec1acf1bf9d4c52dbcd40d68de1b8d1be285fd6d4bd6f388003cc5c35cb6a4"
Sep 11 21:57:29.049745 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:29.049710    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ec1acf1bf9d4c52dbcd40d68de1b8d1be285fd6d4bd6f388003cc5c35cb6a4"} err="failed to get container status \"75ec1acf1bf9d4c52dbcd40d68de1b8d1be285fd6d4bd6f388003cc5c35cb6a4\": rpc error: code = NotFound desc = could not find container \"75ec1acf1bf9d4c52dbcd40d68de1b8d1be285fd6d4bd6f388003cc5c35cb6a4\": container with ID starting with 75ec1acf1bf9d4c52dbcd40d68de1b8d1be285fd6d4bd6f388003cc5c35cb6a4 not found: ID does not exist"
Sep 11 21:57:29.336561 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:29.336481    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07d3dee0-485f-4229-b5c2-5994feb13679" path="/var/lib/kubelet/pods/07d3dee0-485f-4229-b5c2-5994feb13679/volumes"
Sep 11 21:57:30.678532 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:30.678489    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lkxdk" event={"ID":"c57c2e4e-3ab4-4276-9016-b9ff4a0202fc","Type":"ContainerStarted","Data":"431b00e8bf600c769e68a23b61b9c16a8678372697bd8490ae7f79637ae46174"}
Sep 11 21:57:30.678998 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:30.678553    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lkxdk"
Sep 11 21:57:30.680081 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:30.680056    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rlw6w" event={"ID":"c6895de2-3e55-4c67-8107-7ed453c7a934","Type":"ContainerStarted","Data":"b3acc56fca9ce83fc9038ad3b164c98af948db361b37391cd33b1b8046780321"}
Sep 11 21:57:30.680168 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:30.680114    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rlw6w"
Sep 11 21:57:30.719189 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:30.719140    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lkxdk" podStartSLOduration=2.924203028 podStartE2EDuration="5.719122483s" totalImagesPullingTime="2.794919455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:57:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:57:30.718195688 +0000 UTC m=+303.893272058" watchObservedRunningTime="2026-09-11 21:57:30.719122483 +0000 UTC m=+303.894198900"
Sep 11 21:57:30.753774 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:57:30.753720    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rlw6w" podStartSLOduration=2.940395251 podStartE2EDuration="5.753701657s" totalImagesPullingTime="2.813306406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:57:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:57:30.753038635 +0000 UTC m=+303.928115026" watchObservedRunningTime="2026-09-11 21:57:30.753701657 +0000 UTC m=+303.928778061"
Sep 11 21:58:01.687097 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:01.687062    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-lkxdk"
Sep 11 21:58:01.690090 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:01.690068    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rlw6w"
Sep 11 21:58:47.360435 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.360400    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d9fbf6d78-f9vx9"]
Sep 11 21:58:47.360889 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.360695    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07d3dee0-485f-4229-b5c2-5994feb13679" containerName="discovery"
Sep 11 21:58:47.360889 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.360707    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="07d3dee0-485f-4229-b5c2-5994feb13679" containerName="discovery"
Sep 11 21:58:47.360889 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.360757    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="07d3dee0-485f-4229-b5c2-5994feb13679" containerName="discovery"
Sep 11 21:58:47.364845 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.364823    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:47.375819 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.375799    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d9fbf6d78-f9vx9"]
Sep 11 21:58:47.456445 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.456399    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4d1f5cc-f268-46f6-8cce-b126370c7def-console-serving-cert\") pod \"console-5d9fbf6d78-f9vx9\" (UID: \"e4d1f5cc-f268-46f6-8cce-b126370c7def\") " pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:47.456619 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.456468    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d1f5cc-f268-46f6-8cce-b126370c7def-service-ca\") pod \"console-5d9fbf6d78-f9vx9\" (UID: \"e4d1f5cc-f268-46f6-8cce-b126370c7def\") " pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:47.456619 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.456602    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4d1f5cc-f268-46f6-8cce-b126370c7def-oauth-serving-cert\") pod \"console-5d9fbf6d78-f9vx9\" (UID: \"e4d1f5cc-f268-46f6-8cce-b126370c7def\") " pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:47.456706 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.456636    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4d1f5cc-f268-46f6-8cce-b126370c7def-console-config\") pod \"console-5d9fbf6d78-f9vx9\" (UID: \"e4d1f5cc-f268-46f6-8cce-b126370c7def\") " pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:47.456706 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.456675    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4d1f5cc-f268-46f6-8cce-b126370c7def-trusted-ca-bundle\") pod \"console-5d9fbf6d78-f9vx9\" (UID: \"e4d1f5cc-f268-46f6-8cce-b126370c7def\") " pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:47.456781 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.456703    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h96sv\" (UniqueName: \"kubernetes.io/projected/e4d1f5cc-f268-46f6-8cce-b126370c7def-kube-api-access-h96sv\") pod \"console-5d9fbf6d78-f9vx9\" (UID: \"e4d1f5cc-f268-46f6-8cce-b126370c7def\") " pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:47.456781 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.456741    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4d1f5cc-f268-46f6-8cce-b126370c7def-console-oauth-config\") pod \"console-5d9fbf6d78-f9vx9\" (UID: \"e4d1f5cc-f268-46f6-8cce-b126370c7def\") " pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:47.557825 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.557780    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4d1f5cc-f268-46f6-8cce-b126370c7def-console-serving-cert\") pod \"console-5d9fbf6d78-f9vx9\" (UID: \"e4d1f5cc-f268-46f6-8cce-b126370c7def\") " pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:47.558029 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.557841    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d1f5cc-f268-46f6-8cce-b126370c7def-service-ca\") pod \"console-5d9fbf6d78-f9vx9\" (UID: \"e4d1f5cc-f268-46f6-8cce-b126370c7def\") " pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:47.558029 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.557906    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4d1f5cc-f268-46f6-8cce-b126370c7def-oauth-serving-cert\") pod \"console-5d9fbf6d78-f9vx9\" (UID: \"e4d1f5cc-f268-46f6-8cce-b126370c7def\") " pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:47.558029 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.557926    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4d1f5cc-f268-46f6-8cce-b126370c7def-console-config\") pod \"console-5d9fbf6d78-f9vx9\" (UID: \"e4d1f5cc-f268-46f6-8cce-b126370c7def\") " pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:47.558029 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.557957    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4d1f5cc-f268-46f6-8cce-b126370c7def-trusted-ca-bundle\") pod \"console-5d9fbf6d78-f9vx9\" (UID: \"e4d1f5cc-f268-46f6-8cce-b126370c7def\") " pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:47.558029 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.557980    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h96sv\" (UniqueName: \"kubernetes.io/projected/e4d1f5cc-f268-46f6-8cce-b126370c7def-kube-api-access-h96sv\") pod \"console-5d9fbf6d78-f9vx9\" (UID: \"e4d1f5cc-f268-46f6-8cce-b126370c7def\") " pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:47.558029 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.558005    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4d1f5cc-f268-46f6-8cce-b126370c7def-console-oauth-config\") pod \"console-5d9fbf6d78-f9vx9\" (UID: \"e4d1f5cc-f268-46f6-8cce-b126370c7def\") " pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:47.558785 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.558758    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d1f5cc-f268-46f6-8cce-b126370c7def-service-ca\") pod \"console-5d9fbf6d78-f9vx9\" (UID: \"e4d1f5cc-f268-46f6-8cce-b126370c7def\") " pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:47.558914 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.558788    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e4d1f5cc-f268-46f6-8cce-b126370c7def-oauth-serving-cert\") pod \"console-5d9fbf6d78-f9vx9\" (UID: \"e4d1f5cc-f268-46f6-8cce-b126370c7def\") " pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:47.558914 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.558813    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e4d1f5cc-f268-46f6-8cce-b126370c7def-console-config\") pod \"console-5d9fbf6d78-f9vx9\" (UID: \"e4d1f5cc-f268-46f6-8cce-b126370c7def\") " pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:47.558914 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.558888    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4d1f5cc-f268-46f6-8cce-b126370c7def-trusted-ca-bundle\") pod \"console-5d9fbf6d78-f9vx9\" (UID: \"e4d1f5cc-f268-46f6-8cce-b126370c7def\") " pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:47.560427 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.560398    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e4d1f5cc-f268-46f6-8cce-b126370c7def-console-serving-cert\") pod \"console-5d9fbf6d78-f9vx9\" (UID: \"e4d1f5cc-f268-46f6-8cce-b126370c7def\") " pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:47.560518 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.560450    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e4d1f5cc-f268-46f6-8cce-b126370c7def-console-oauth-config\") pod \"console-5d9fbf6d78-f9vx9\" (UID: \"e4d1f5cc-f268-46f6-8cce-b126370c7def\") " pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:47.566561 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.566544    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h96sv\" (UniqueName: \"kubernetes.io/projected/e4d1f5cc-f268-46f6-8cce-b126370c7def-kube-api-access-h96sv\") pod \"console-5d9fbf6d78-f9vx9\" (UID: \"e4d1f5cc-f268-46f6-8cce-b126370c7def\") " pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:47.674109 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.674071    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:47.947883 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:47.947813    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d9fbf6d78-f9vx9"]
Sep 11 21:58:47.950862 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:58:47.950834    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d1f5cc_f268_46f6_8cce_b126370c7def.slice/crio-1b63d3a040c996860917ae380d2a87e7394792416f80043ba7145a3f0b7682c8 WatchSource:0}: Error finding container 1b63d3a040c996860917ae380d2a87e7394792416f80043ba7145a3f0b7682c8: Status 404 returned error can't find the container with id 1b63d3a040c996860917ae380d2a87e7394792416f80043ba7145a3f0b7682c8
Sep 11 21:58:48.011604 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:48.011577    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9fbf6d78-f9vx9" event={"ID":"e4d1f5cc-f268-46f6-8cce-b126370c7def","Type":"ContainerStarted","Data":"1b63d3a040c996860917ae380d2a87e7394792416f80043ba7145a3f0b7682c8"}
Sep 11 21:58:49.020543 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:49.020508    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9fbf6d78-f9vx9" event={"ID":"e4d1f5cc-f268-46f6-8cce-b126370c7def","Type":"ContainerStarted","Data":"ab4f81660b045904ead306b10033bdf951e8ffca738784d41b63c070733d46ec"}
Sep 11 21:58:49.041279 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:49.041206    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d9fbf6d78-f9vx9" podStartSLOduration=2.041188564 podStartE2EDuration="2.041188564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:58:49.040255464 +0000 UTC m=+382.215331858" watchObservedRunningTime="2026-09-11 21:58:49.041188564 +0000 UTC m=+382.216264935"
Sep 11 21:58:56.972748 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:56.972712    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"]
Sep 11 21:58:56.976014 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:56.975996    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"
Sep 11 21:58:56.978975 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:56.978951    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 21:58:56.978975 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:56.978950    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 21:58:56.979143 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:56.979035    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8bxkr\""
Sep 11 21:58:56.990791 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:56.990771    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"]
Sep 11 21:58:57.142580 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:57.142538    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sqv7\" (UniqueName: \"kubernetes.io/projected/4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e-kube-api-access-8sqv7\") pod \"kuadrant-operator-controller-manager-76f8f686bf-m4qsq\" (UID: \"4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"
Sep 11 21:58:57.142796 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:57.142606    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-m4qsq\" (UID: \"4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"
Sep 11 21:58:57.243099 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:57.243019    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sqv7\" (UniqueName: \"kubernetes.io/projected/4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e-kube-api-access-8sqv7\") pod \"kuadrant-operator-controller-manager-76f8f686bf-m4qsq\" (UID: \"4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"
Sep 11 21:58:57.243099 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:57.243081    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-m4qsq\" (UID: \"4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"
Sep 11 21:58:57.243522 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:57.243504    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-m4qsq\" (UID: \"4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"
Sep 11 21:58:57.264184 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:57.264164    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sqv7\" (UniqueName: \"kubernetes.io/projected/4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e-kube-api-access-8sqv7\") pod \"kuadrant-operator-controller-manager-76f8f686bf-m4qsq\" (UID: \"4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"
Sep 11 21:58:57.286138 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:57.286114    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"
Sep 11 21:58:57.562524 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:57.562448    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"]
Sep 11 21:58:57.566745 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:58:57.566716    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8fc9db_b58d_4fec_9c2f_d68d92a38c9e.slice/crio-2336d0e716b67cb1eb999ee03d48dee470c4f1dc016903571d1e75fcf1b4bbb8 WatchSource:0}: Error finding container 2336d0e716b67cb1eb999ee03d48dee470c4f1dc016903571d1e75fcf1b4bbb8: Status 404 returned error can't find the container with id 2336d0e716b67cb1eb999ee03d48dee470c4f1dc016903571d1e75fcf1b4bbb8
Sep 11 21:58:57.569731 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:57.569711    2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 21:58:57.674441 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:57.674407    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:57.674651 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:57.674635    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:57.680468 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:57.680445    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:58.064518 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:58.064478    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq" event={"ID":"4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e","Type":"ContainerStarted","Data":"2336d0e716b67cb1eb999ee03d48dee470c4f1dc016903571d1e75fcf1b4bbb8"}
Sep 11 21:58:58.068919 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:58.068894    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d9fbf6d78-f9vx9"
Sep 11 21:58:58.120778 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:58.120741    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74996bdd8f-x8pfn"]
Sep 11 21:58:58.987773 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:58.987734    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rcj2k"]
Sep 11 21:58:58.991248 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:58.991220    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rcj2k"
Sep 11 21:58:58.994462 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:58.994400    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 11 21:58:58.994462 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:58.994459    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-27fc2\""
Sep 11 21:58:59.005835 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:59.005746    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rcj2k"]
Sep 11 21:58:59.163543 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:59.163507    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zng9l\" (UniqueName: \"kubernetes.io/projected/77cee7bd-677f-4371-982a-49c059de9b54-kube-api-access-zng9l\") pod \"dns-operator-controller-manager-7b84cd7c9c-rcj2k\" (UID: \"77cee7bd-677f-4371-982a-49c059de9b54\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rcj2k"
Sep 11 21:58:59.264996 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:59.264896    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zng9l\" (UniqueName: \"kubernetes.io/projected/77cee7bd-677f-4371-982a-49c059de9b54-kube-api-access-zng9l\") pod \"dns-operator-controller-manager-7b84cd7c9c-rcj2k\" (UID: \"77cee7bd-677f-4371-982a-49c059de9b54\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rcj2k"
Sep 11 21:58:59.274260 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:59.274223    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zng9l\" (UniqueName: \"kubernetes.io/projected/77cee7bd-677f-4371-982a-49c059de9b54-kube-api-access-zng9l\") pod \"dns-operator-controller-manager-7b84cd7c9c-rcj2k\" (UID: \"77cee7bd-677f-4371-982a-49c059de9b54\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rcj2k"
Sep 11 21:58:59.305165 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:58:59.305127    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rcj2k"
Sep 11 21:59:00.251076 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:00.251048    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rcj2k"]
Sep 11 21:59:00.253635 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:59:00.253605    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77cee7bd_677f_4371_982a_49c059de9b54.slice/crio-41632c83f9ec255978e6d96300aaf583853b61486b55b3e85c605a287237f119 WatchSource:0}: Error finding container 41632c83f9ec255978e6d96300aaf583853b61486b55b3e85c605a287237f119: Status 404 returned error can't find the container with id 41632c83f9ec255978e6d96300aaf583853b61486b55b3e85c605a287237f119
Sep 11 21:59:01.083753 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:01.083714    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rcj2k" event={"ID":"77cee7bd-677f-4371-982a-49c059de9b54","Type":"ContainerStarted","Data":"41632c83f9ec255978e6d96300aaf583853b61486b55b3e85c605a287237f119"}
Sep 11 21:59:01.194189 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:01.194143    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dxrrp"]
Sep 11 21:59:01.198010 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:01.197984    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dxrrp"
Sep 11 21:59:01.200881 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:01.200839    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-8jdwz\""
Sep 11 21:59:01.211517 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:01.211493    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dxrrp"]
Sep 11 21:59:01.385946 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:01.385798    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dhh7\" (UniqueName: \"kubernetes.io/projected/b0c4d72a-eb03-4b1a-9709-3351d7827036-kube-api-access-7dhh7\") pod \"limitador-operator-controller-manager-74495c69f8-dxrrp\" (UID: \"b0c4d72a-eb03-4b1a-9709-3351d7827036\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dxrrp"
Sep 11 21:59:01.487818 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:01.487771    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dhh7\" (UniqueName: \"kubernetes.io/projected/b0c4d72a-eb03-4b1a-9709-3351d7827036-kube-api-access-7dhh7\") pod \"limitador-operator-controller-manager-74495c69f8-dxrrp\" (UID: \"b0c4d72a-eb03-4b1a-9709-3351d7827036\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dxrrp"
Sep 11 21:59:01.502434 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:01.502405    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dhh7\" (UniqueName: \"kubernetes.io/projected/b0c4d72a-eb03-4b1a-9709-3351d7827036-kube-api-access-7dhh7\") pod \"limitador-operator-controller-manager-74495c69f8-dxrrp\" (UID: \"b0c4d72a-eb03-4b1a-9709-3351d7827036\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dxrrp"
Sep 11 21:59:01.512171 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:01.512139    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dxrrp"
Sep 11 21:59:02.263857 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:02.263770    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dxrrp"]
Sep 11 21:59:02.268663 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:59:02.268622    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c4d72a_eb03_4b1a_9709_3351d7827036.slice/crio-8776de472e998e553f2fdc9cfc52cf09189dd95c1c585b384d024e38c023cf82 WatchSource:0}: Error finding container 8776de472e998e553f2fdc9cfc52cf09189dd95c1c585b384d024e38c023cf82: Status 404 returned error can't find the container with id 8776de472e998e553f2fdc9cfc52cf09189dd95c1c585b384d024e38c023cf82
Sep 11 21:59:03.096703 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:03.096668    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq" event={"ID":"4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e","Type":"ContainerStarted","Data":"df758748ca856e9d50f2b5b1eb31120d7d8cac2a16b642c9c5ebf48cb2986d10"}
Sep 11 21:59:03.097161 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:03.096791    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"
Sep 11 21:59:03.098399 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:03.098371    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dxrrp" event={"ID":"b0c4d72a-eb03-4b1a-9709-3351d7827036","Type":"ContainerStarted","Data":"8776de472e998e553f2fdc9cfc52cf09189dd95c1c585b384d024e38c023cf82"}
Sep 11 21:59:03.100375 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:03.100350    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rcj2k" event={"ID":"77cee7bd-677f-4371-982a-49c059de9b54","Type":"ContainerStarted","Data":"d7a67f09722187a68dab270bcfb2392a2d84f9e74c774f4a0736fd472ab7f658"}
Sep 11 21:59:03.100494 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:03.100404    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rcj2k"
Sep 11 21:59:03.121489 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:03.121213    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq" podStartSLOduration=2.6340510520000002 podStartE2EDuration="7.121189758s" totalImagesPullingTime="4.487138706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:58:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:59:03.118202153 +0000 UTC m=+396.293278543" watchObservedRunningTime="2026-09-11 21:59:03.121189758 +0000 UTC m=+396.296266130"
Sep 11 21:59:03.140940 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:03.140884    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rcj2k" podStartSLOduration=2.454199235 podStartE2EDuration="5.140861788s" totalImagesPullingTime="2.686662553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:59:03.139779864 +0000 UTC m=+396.314856259" watchObservedRunningTime="2026-09-11 21:59:03.140861788 +0000 UTC m=+396.315938160"
Sep 11 21:59:04.107017 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:04.106921    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dxrrp" event={"ID":"b0c4d72a-eb03-4b1a-9709-3351d7827036","Type":"ContainerStarted","Data":"00596e1d6fc8555e2c0c80d42d8edbba1b3af1bbbe09f1d2fd4fb25315630004"}
Sep 11 21:59:04.107464 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:04.107018    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dxrrp"
Sep 11 21:59:04.125440 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:04.125391    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dxrrp" podStartSLOduration=1.57927157 podStartE2EDuration="3.125372626s" totalImagesPullingTime="1.546101056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:59:04.123078888 +0000 UTC m=+397.298155280" watchObservedRunningTime="2026-09-11 21:59:04.125372626 +0000 UTC m=+397.300448998"
Sep 11 21:59:04.200949 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:04.200918    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-mcqjh"]
Sep 11 21:59:04.204336 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:04.204311    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-mcqjh"
Sep 11 21:59:04.207416 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:04.207395    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-vw78v\""
Sep 11 21:59:04.219414 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:04.219391    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-mcqjh"]
Sep 11 21:59:04.317085 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:04.317042    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvc7b\" (UniqueName: \"kubernetes.io/projected/096c0792-1d86-45fc-a9e3-ecfc43321e48-kube-api-access-kvc7b\") pod \"authorino-operator-76b478bd78-mcqjh\" (UID: \"096c0792-1d86-45fc-a9e3-ecfc43321e48\") " pod="kuadrant-system/authorino-operator-76b478bd78-mcqjh"
Sep 11 21:59:04.417710 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:04.417676    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvc7b\" (UniqueName: \"kubernetes.io/projected/096c0792-1d86-45fc-a9e3-ecfc43321e48-kube-api-access-kvc7b\") pod \"authorino-operator-76b478bd78-mcqjh\" (UID: \"096c0792-1d86-45fc-a9e3-ecfc43321e48\") " pod="kuadrant-system/authorino-operator-76b478bd78-mcqjh"
Sep 11 21:59:04.426216 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:04.426181    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvc7b\" (UniqueName: \"kubernetes.io/projected/096c0792-1d86-45fc-a9e3-ecfc43321e48-kube-api-access-kvc7b\") pod \"authorino-operator-76b478bd78-mcqjh\" (UID: \"096c0792-1d86-45fc-a9e3-ecfc43321e48\") " pod="kuadrant-system/authorino-operator-76b478bd78-mcqjh"
Sep 11 21:59:04.513495 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:04.513463    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-mcqjh"
Sep 11 21:59:04.787621 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:04.787597    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-mcqjh"]
Sep 11 21:59:04.790130 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:59:04.790099    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096c0792_1d86_45fc_a9e3_ecfc43321e48.slice/crio-194eb1ec796f6d2d9e22b9eb4f52e9129106764e549f79d1db591de2aca2e1c9 WatchSource:0}: Error finding container 194eb1ec796f6d2d9e22b9eb4f52e9129106764e549f79d1db591de2aca2e1c9: Status 404 returned error can't find the container with id 194eb1ec796f6d2d9e22b9eb4f52e9129106764e549f79d1db591de2aca2e1c9
Sep 11 21:59:05.054357 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:05.054262    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-64t4p"]
Sep 11 21:59:05.059170 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:05.059142    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-64t4p"
Sep 11 21:59:05.061797 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:05.061776    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-k94pm\""
Sep 11 21:59:05.061899 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:05.061840    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 11 21:59:05.062040 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:05.062026    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 11 21:59:05.065300 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:05.065276    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-64t4p"]
Sep 11 21:59:05.113367 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:05.113317    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-mcqjh" event={"ID":"096c0792-1d86-45fc-a9e3-ecfc43321e48","Type":"ContainerStarted","Data":"194eb1ec796f6d2d9e22b9eb4f52e9129106764e549f79d1db591de2aca2e1c9"}
Sep 11 21:59:05.224478 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:05.224445    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/06b4da6f-ba91-40be-8cb9-12c20797c691-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-64t4p\" (UID: \"06b4da6f-ba91-40be-8cb9-12c20797c691\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-64t4p"
Sep 11 21:59:05.224668 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:05.224511    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfwlm\" (UniqueName: \"kubernetes.io/projected/06b4da6f-ba91-40be-8cb9-12c20797c691-kube-api-access-sfwlm\") pod \"kuadrant-console-plugin-fbdd4f445-64t4p\" (UID: \"06b4da6f-ba91-40be-8cb9-12c20797c691\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-64t4p"
Sep 11 21:59:05.224668 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:05.224545    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/06b4da6f-ba91-40be-8cb9-12c20797c691-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-64t4p\" (UID: \"06b4da6f-ba91-40be-8cb9-12c20797c691\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-64t4p"
Sep 11 21:59:05.325616 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:05.325527    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/06b4da6f-ba91-40be-8cb9-12c20797c691-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-64t4p\" (UID: \"06b4da6f-ba91-40be-8cb9-12c20797c691\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-64t4p"
Sep 11 21:59:05.325616 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:05.325578    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfwlm\" (UniqueName: \"kubernetes.io/projected/06b4da6f-ba91-40be-8cb9-12c20797c691-kube-api-access-sfwlm\") pod \"kuadrant-console-plugin-fbdd4f445-64t4p\" (UID: \"06b4da6f-ba91-40be-8cb9-12c20797c691\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-64t4p"
Sep 11 21:59:05.325824 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:05.325703    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/06b4da6f-ba91-40be-8cb9-12c20797c691-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-64t4p\" (UID: \"06b4da6f-ba91-40be-8cb9-12c20797c691\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-64t4p"
Sep 11 21:59:05.325868 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:59:05.325823    2587 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 11 21:59:05.325904 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:59:05.325890    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06b4da6f-ba91-40be-8cb9-12c20797c691-plugin-serving-cert podName:06b4da6f-ba91-40be-8cb9-12c20797c691 nodeName:}" failed. No retries permitted until 2026-09-11 21:59:05.825872325 +0000 UTC m=+399.000948673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/06b4da6f-ba91-40be-8cb9-12c20797c691-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-64t4p" (UID: "06b4da6f-ba91-40be-8cb9-12c20797c691") : secret "plugin-serving-cert" not found
Sep 11 21:59:05.326296 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:05.326276    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/06b4da6f-ba91-40be-8cb9-12c20797c691-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-64t4p\" (UID: \"06b4da6f-ba91-40be-8cb9-12c20797c691\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-64t4p"
Sep 11 21:59:05.337404 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:05.337370    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfwlm\" (UniqueName: \"kubernetes.io/projected/06b4da6f-ba91-40be-8cb9-12c20797c691-kube-api-access-sfwlm\") pod \"kuadrant-console-plugin-fbdd4f445-64t4p\" (UID: \"06b4da6f-ba91-40be-8cb9-12c20797c691\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-64t4p"
Sep 11 21:59:05.830276 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:05.830240    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/06b4da6f-ba91-40be-8cb9-12c20797c691-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-64t4p\" (UID: \"06b4da6f-ba91-40be-8cb9-12c20797c691\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-64t4p"
Sep 11 21:59:05.833119 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:05.833091    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/06b4da6f-ba91-40be-8cb9-12c20797c691-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-64t4p\" (UID: \"06b4da6f-ba91-40be-8cb9-12c20797c691\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-64t4p"
Sep 11 21:59:05.970531 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:05.970492    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-64t4p"
Sep 11 21:59:06.252752 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:06.252666    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-64t4p"]
Sep 11 21:59:06.352581 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:59:06.352541    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06b4da6f_ba91_40be_8cb9_12c20797c691.slice/crio-0d43c3cd8536bdf980da222c1150872b8ff14baec0c787b8d4f0b17ba3fdcd9b WatchSource:0}: Error finding container 0d43c3cd8536bdf980da222c1150872b8ff14baec0c787b8d4f0b17ba3fdcd9b: Status 404 returned error can't find the container with id 0d43c3cd8536bdf980da222c1150872b8ff14baec0c787b8d4f0b17ba3fdcd9b
Sep 11 21:59:07.130379 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:07.130321    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-64t4p" event={"ID":"06b4da6f-ba91-40be-8cb9-12c20797c691","Type":"ContainerStarted","Data":"0d43c3cd8536bdf980da222c1150872b8ff14baec0c787b8d4f0b17ba3fdcd9b"}
Sep 11 21:59:07.132164 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:07.132134    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-mcqjh" event={"ID":"096c0792-1d86-45fc-a9e3-ecfc43321e48","Type":"ContainerStarted","Data":"869443dc658bc01be0a0e245452b010c10694a6ec0bd0fc801819974b71ad8c3"}
Sep 11 21:59:07.132302 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:07.132280    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-mcqjh"
Sep 11 21:59:07.150381 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:07.150321    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-mcqjh" podStartSLOduration=1.527234987 podStartE2EDuration="3.150303545s" totalImagesPullingTime="1.623068558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:59:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:59:07.149836434 +0000 UTC m=+400.324912825" watchObservedRunningTime="2026-09-11 21:59:07.150303545 +0000 UTC m=+400.325379917"
Sep 11 21:59:11.155742 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:11.155708    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-64t4p" event={"ID":"06b4da6f-ba91-40be-8cb9-12c20797c691","Type":"ContainerStarted","Data":"d347259f33c8d7a529d102bcf5608c57e8753d5ca0d862557cfdcda0f8e39f5e"}
Sep 11 21:59:11.171622 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:11.171568    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-64t4p" podStartSLOduration=1.4892517029999999 podStartE2EDuration="6.171545008s" totalImagesPullingTime="4.682293305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:59:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:59:11.1712352 +0000 UTC m=+404.346311570" watchObservedRunningTime="2026-09-11 21:59:11.171545008 +0000 UTC m=+404.346621377"
Sep 11 21:59:14.110044 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:14.110009    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rcj2k"
Sep 11 21:59:14.110415 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:14.110171    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"
Sep 11 21:59:15.115670 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:15.115633    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dxrrp"
Sep 11 21:59:16.174684 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.174648    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-zj46d"]
Sep 11 21:59:16.178430 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.178408    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-zj46d"
Sep 11 21:59:16.193391 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.193357    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-zj46d"]
Sep 11 21:59:16.219446 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.219413    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7lvf\" (UniqueName: \"kubernetes.io/projected/1419f007-66b6-44f4-a174-66271f6b3f44-kube-api-access-f7lvf\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-zj46d\" (UID: \"1419f007-66b6-44f4-a174-66271f6b3f44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-zj46d"
Sep 11 21:59:16.219583 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.219452    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1419f007-66b6-44f4-a174-66271f6b3f44-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-zj46d\" (UID: \"1419f007-66b6-44f4-a174-66271f6b3f44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-zj46d"
Sep 11 21:59:16.281156 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.281124    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-zj46d"]
Sep 11 21:59:16.281433 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:59:16.281412    2587 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-f7lvf], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-zj46d" podUID="1419f007-66b6-44f4-a174-66271f6b3f44"
Sep 11 21:59:16.320894 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.320860    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7lvf\" (UniqueName: \"kubernetes.io/projected/1419f007-66b6-44f4-a174-66271f6b3f44-kube-api-access-f7lvf\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-zj46d\" (UID: \"1419f007-66b6-44f4-a174-66271f6b3f44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-zj46d"
Sep 11 21:59:16.320894 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.320900    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1419f007-66b6-44f4-a174-66271f6b3f44-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-zj46d\" (UID: \"1419f007-66b6-44f4-a174-66271f6b3f44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-zj46d"
Sep 11 21:59:16.321297 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.321278    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1419f007-66b6-44f4-a174-66271f6b3f44-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-zj46d\" (UID: \"1419f007-66b6-44f4-a174-66271f6b3f44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-zj46d"
Sep 11 21:59:16.341720 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.341693    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7lvf\" (UniqueName: \"kubernetes.io/projected/1419f007-66b6-44f4-a174-66271f6b3f44-kube-api-access-f7lvf\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-zj46d\" (UID: \"1419f007-66b6-44f4-a174-66271f6b3f44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-zj46d"
Sep 11 21:59:16.385174 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.385138    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"]
Sep 11 21:59:16.385407 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.385382    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq" podUID="4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e" containerName="manager" containerID="cri-o://df758748ca856e9d50f2b5b1eb31120d7d8cac2a16b642c9c5ebf48cb2986d10" gracePeriod=2
Sep 11 21:59:16.413034 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.413008    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"]
Sep 11 21:59:16.429435 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:59:16.429360    2587 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-m4qsq\": pods \"kuadrant-operator-controller-manager-76f8f686bf-m4qsq\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"
Sep 11 21:59:16.433570 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.433539    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-zj46d"]
Sep 11 21:59:16.454416 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.454391    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtff9"]
Sep 11 21:59:16.454731 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.454719    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e" containerName="manager"
Sep 11 21:59:16.454782 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.454732    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e" containerName="manager"
Sep 11 21:59:16.454816 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.454790    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e" containerName="manager"
Sep 11 21:59:16.457718 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.457694    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtff9"
Sep 11 21:59:16.459030 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.459009    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-zj46d"]
Sep 11 21:59:16.462311 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:59:16.462281    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-m4qsq\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"
Sep 11 21:59:16.470674 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.470649    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dxrrp"]
Sep 11 21:59:16.470873 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.470846    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dxrrp" podUID="b0c4d72a-eb03-4b1a-9709-3351d7827036" containerName="manager" containerID="cri-o://00596e1d6fc8555e2c0c80d42d8edbba1b3af1bbbe09f1d2fd4fb25315630004" gracePeriod=2
Sep 11 21:59:16.491222 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.491179    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtff9"]
Sep 11 21:59:16.492941 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.492919    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dxrrp"]
Sep 11 21:59:16.522546 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.522508    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mvmv\" (UniqueName: \"kubernetes.io/projected/69f007b1-9538-4d01-9513-d98e88d1f52e-kube-api-access-5mvmv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wtff9\" (UID: \"69f007b1-9538-4d01-9513-d98e88d1f52e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtff9"
Sep 11 21:59:16.522696 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.522624    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69f007b1-9538-4d01-9513-d98e88d1f52e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wtff9\" (UID: \"69f007b1-9538-4d01-9513-d98e88d1f52e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtff9"
Sep 11 21:59:16.526761 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.526733    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-vjrz7"]
Sep 11 21:59:16.527170 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.527151    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0c4d72a-eb03-4b1a-9709-3351d7827036" containerName="manager"
Sep 11 21:59:16.527275 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.527172    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0c4d72a-eb03-4b1a-9709-3351d7827036" containerName="manager"
Sep 11 21:59:16.527275 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.527259    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0c4d72a-eb03-4b1a-9709-3351d7827036" containerName="manager"
Sep 11 21:59:16.530290 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.530269    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vjrz7"
Sep 11 21:59:16.542078 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:59:16.542047    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-dxrrp\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="b0c4d72a-eb03-4b1a-9709-3351d7827036" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dxrrp"
Sep 11 21:59:16.544893 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:59:16.544864    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-m4qsq\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"
Sep 11 21:59:16.549236 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.549212    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-vjrz7"]
Sep 11 21:59:16.623167 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.623097    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69f007b1-9538-4d01-9513-d98e88d1f52e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wtff9\" (UID: \"69f007b1-9538-4d01-9513-d98e88d1f52e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtff9"
Sep 11 21:59:16.623462 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.623178    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mvmv\" (UniqueName: \"kubernetes.io/projected/69f007b1-9538-4d01-9513-d98e88d1f52e-kube-api-access-5mvmv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wtff9\" (UID: \"69f007b1-9538-4d01-9513-d98e88d1f52e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtff9"
Sep 11 21:59:16.623462 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.623232    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqjrd\" (UniqueName: \"kubernetes.io/projected/851204ff-15b4-40e8-9558-1d6fd04cfabf-kube-api-access-jqjrd\") pod \"limitador-operator-controller-manager-74495c69f8-vjrz7\" (UID: \"851204ff-15b4-40e8-9558-1d6fd04cfabf\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vjrz7"
Sep 11 21:59:16.623612 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.623516    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69f007b1-9538-4d01-9513-d98e88d1f52e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wtff9\" (UID: \"69f007b1-9538-4d01-9513-d98e88d1f52e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtff9"
Sep 11 21:59:16.636933 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.636876    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mvmv\" (UniqueName: \"kubernetes.io/projected/69f007b1-9538-4d01-9513-d98e88d1f52e-kube-api-access-5mvmv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wtff9\" (UID: \"69f007b1-9538-4d01-9513-d98e88d1f52e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtff9"
Sep 11 21:59:16.701796 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.701774    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"
Sep 11 21:59:16.704455 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:59:16.704420    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-dxrrp\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="b0c4d72a-eb03-4b1a-9709-3351d7827036" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dxrrp"
Sep 11 21:59:16.707189 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:59:16.707157    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-m4qsq\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"
Sep 11 21:59:16.707995 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.707980    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dxrrp"
Sep 11 21:59:16.710409 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:59:16.710385    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-m4qsq\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"
Sep 11 21:59:16.712804 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:59:16.712783    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-dxrrp\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="b0c4d72a-eb03-4b1a-9709-3351d7827036" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dxrrp"
Sep 11 21:59:16.724230 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.724212    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sqv7\" (UniqueName: \"kubernetes.io/projected/4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e-kube-api-access-8sqv7\") pod \"4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e\" (UID: \"4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e\") "
Sep 11 21:59:16.724307 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.724284    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e-extensions-socket-volume\") pod \"4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e\" (UID: \"4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e\") "
Sep 11 21:59:16.724422 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.724406    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqjrd\" (UniqueName: \"kubernetes.io/projected/851204ff-15b4-40e8-9558-1d6fd04cfabf-kube-api-access-jqjrd\") pod \"limitador-operator-controller-manager-74495c69f8-vjrz7\" (UID: \"851204ff-15b4-40e8-9558-1d6fd04cfabf\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vjrz7"
Sep 11 21:59:16.724746 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.724723    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e" (UID: "4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:59:16.726369 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.726320    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e-kube-api-access-8sqv7" (OuterVolumeSpecName: "kube-api-access-8sqv7") pod "4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e" (UID: "4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e"). InnerVolumeSpecName "kube-api-access-8sqv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:59:16.733181 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.733158    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqjrd\" (UniqueName: \"kubernetes.io/projected/851204ff-15b4-40e8-9558-1d6fd04cfabf-kube-api-access-jqjrd\") pod \"limitador-operator-controller-manager-74495c69f8-vjrz7\" (UID: \"851204ff-15b4-40e8-9558-1d6fd04cfabf\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vjrz7"
Sep 11 21:59:16.777836 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.777799    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtff9"
Sep 11 21:59:16.825511 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.825479    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dhh7\" (UniqueName: \"kubernetes.io/projected/b0c4d72a-eb03-4b1a-9709-3351d7827036-kube-api-access-7dhh7\") pod \"b0c4d72a-eb03-4b1a-9709-3351d7827036\" (UID: \"b0c4d72a-eb03-4b1a-9709-3351d7827036\") "
Sep 11 21:59:16.825790 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.825769    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sqv7\" (UniqueName: \"kubernetes.io/projected/4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e-kube-api-access-8sqv7\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:59:16.825870 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.825795    2587 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e-extensions-socket-volume\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:59:16.828023 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.827991    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c4d72a-eb03-4b1a-9709-3351d7827036-kube-api-access-7dhh7" (OuterVolumeSpecName: "kube-api-access-7dhh7") pod "b0c4d72a-eb03-4b1a-9709-3351d7827036" (UID: "b0c4d72a-eb03-4b1a-9709-3351d7827036"). InnerVolumeSpecName "kube-api-access-7dhh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:59:16.852686 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.852652    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vjrz7"
Sep 11 21:59:16.927132 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:16.927104    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dhh7\" (UniqueName: \"kubernetes.io/projected/b0c4d72a-eb03-4b1a-9709-3351d7827036-kube-api-access-7dhh7\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:59:17.061287 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.061264    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtff9"]
Sep 11 21:59:17.063157 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:59:17.063133    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f007b1_9538_4d01_9513_d98e88d1f52e.slice/crio-ac78dd898bdc0e57474bac41161a7815261a4c207ee2889bbb17a2222b7f1839 WatchSource:0}: Error finding container ac78dd898bdc0e57474bac41161a7815261a4c207ee2889bbb17a2222b7f1839: Status 404 returned error can't find the container with id ac78dd898bdc0e57474bac41161a7815261a4c207ee2889bbb17a2222b7f1839
Sep 11 21:59:17.126069 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.126047    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-vjrz7"]
Sep 11 21:59:17.127958 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:59:17.127931    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851204ff_15b4_40e8_9558_1d6fd04cfabf.slice/crio-b1705979bcf51ef63db3fa575abd039db9a9001a3c7abc362b601ad1e29e4783 WatchSource:0}: Error finding container b1705979bcf51ef63db3fa575abd039db9a9001a3c7abc362b601ad1e29e4783: Status 404 returned error can't find the container with id b1705979bcf51ef63db3fa575abd039db9a9001a3c7abc362b601ad1e29e4783
Sep 11 21:59:17.193108 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.193082    2587 generic.go:356] "Generic (PLEG): container finished" podID="4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e" containerID="df758748ca856e9d50f2b5b1eb31120d7d8cac2a16b642c9c5ebf48cb2986d10" exitCode=0
Sep 11 21:59:17.193447 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.193124    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"
Sep 11 21:59:17.193447 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.193135    2587 scope.go:117] "RemoveContainer" containerID="df758748ca856e9d50f2b5b1eb31120d7d8cac2a16b642c9c5ebf48cb2986d10"
Sep 11 21:59:17.194659 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.194636    2587 generic.go:356] "Generic (PLEG): container finished" podID="b0c4d72a-eb03-4b1a-9709-3351d7827036" containerID="00596e1d6fc8555e2c0c80d42d8edbba1b3af1bbbe09f1d2fd4fb25315630004" exitCode=0
Sep 11 21:59:17.194765 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.194679    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dxrrp"
Sep 11 21:59:17.196592 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:59:17.196566    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-dxrrp\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="b0c4d72a-eb03-4b1a-9709-3351d7827036" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dxrrp"
Sep 11 21:59:17.196927 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.196906    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtff9" event={"ID":"69f007b1-9538-4d01-9513-d98e88d1f52e","Type":"ContainerStarted","Data":"15b7b79f2862affaade40430762187c51c890f14fd3d4c4a526e6fba495b40a0"}
Sep 11 21:59:17.196984 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.196946    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtff9" event={"ID":"69f007b1-9538-4d01-9513-d98e88d1f52e","Type":"ContainerStarted","Data":"ac78dd898bdc0e57474bac41161a7815261a4c207ee2889bbb17a2222b7f1839"}
Sep 11 21:59:17.200235 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.197594    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtff9"
Sep 11 21:59:17.202388 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:59:17.201395    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-m4qsq\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"
Sep 11 21:59:17.206707 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.206680    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vjrz7" event={"ID":"851204ff-15b4-40e8-9558-1d6fd04cfabf","Type":"ContainerStarted","Data":"882fdd525f4df6924f33dc55a10ff2622a9e34cc5dd72e0b186dbc4be168a29b"}
Sep 11 21:59:17.206798 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.206732    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vjrz7" event={"ID":"851204ff-15b4-40e8-9558-1d6fd04cfabf","Type":"ContainerStarted","Data":"b1705979bcf51ef63db3fa575abd039db9a9001a3c7abc362b601ad1e29e4783"}
Sep 11 21:59:17.206798 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.206789    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-zj46d"
Sep 11 21:59:17.208979 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.208963    2587 scope.go:117] "RemoveContainer" containerID="df758748ca856e9d50f2b5b1eb31120d7d8cac2a16b642c9c5ebf48cb2986d10"
Sep 11 21:59:17.209274 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:59:17.209250    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df758748ca856e9d50f2b5b1eb31120d7d8cac2a16b642c9c5ebf48cb2986d10\": container with ID starting with df758748ca856e9d50f2b5b1eb31120d7d8cac2a16b642c9c5ebf48cb2986d10 not found: ID does not exist" containerID="df758748ca856e9d50f2b5b1eb31120d7d8cac2a16b642c9c5ebf48cb2986d10"
Sep 11 21:59:17.209342 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.209284    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df758748ca856e9d50f2b5b1eb31120d7d8cac2a16b642c9c5ebf48cb2986d10"} err="failed to get container status \"df758748ca856e9d50f2b5b1eb31120d7d8cac2a16b642c9c5ebf48cb2986d10\": rpc error: code = NotFound desc = could not find container \"df758748ca856e9d50f2b5b1eb31120d7d8cac2a16b642c9c5ebf48cb2986d10\": container with ID starting with df758748ca856e9d50f2b5b1eb31120d7d8cac2a16b642c9c5ebf48cb2986d10 not found: ID does not exist"
Sep 11 21:59:17.209342 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.209307    2587 scope.go:117] "RemoveContainer" containerID="00596e1d6fc8555e2c0c80d42d8edbba1b3af1bbbe09f1d2fd4fb25315630004"
Sep 11 21:59:17.215954 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.215935    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-zj46d"
Sep 11 21:59:17.217203 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.217188    2587 scope.go:117] "RemoveContainer" containerID="00596e1d6fc8555e2c0c80d42d8edbba1b3af1bbbe09f1d2fd4fb25315630004"
Sep 11 21:59:17.217478 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:59:17.217461    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00596e1d6fc8555e2c0c80d42d8edbba1b3af1bbbe09f1d2fd4fb25315630004\": container with ID starting with 00596e1d6fc8555e2c0c80d42d8edbba1b3af1bbbe09f1d2fd4fb25315630004 not found: ID does not exist" containerID="00596e1d6fc8555e2c0c80d42d8edbba1b3af1bbbe09f1d2fd4fb25315630004"
Sep 11 21:59:17.217550 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.217482    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00596e1d6fc8555e2c0c80d42d8edbba1b3af1bbbe09f1d2fd4fb25315630004"} err="failed to get container status \"00596e1d6fc8555e2c0c80d42d8edbba1b3af1bbbe09f1d2fd4fb25315630004\": rpc error: code = NotFound desc = could not find container \"00596e1d6fc8555e2c0c80d42d8edbba1b3af1bbbe09f1d2fd4fb25315630004\": container with ID starting with 00596e1d6fc8555e2c0c80d42d8edbba1b3af1bbbe09f1d2fd4fb25315630004 not found: ID does not exist"
Sep 11 21:59:17.229104 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.229060    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtff9" podStartSLOduration=1.22904487 podStartE2EDuration="1.22904487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:59:17.227994571 +0000 UTC m=+410.403070963" watchObservedRunningTime="2026-09-11 21:59:17.22904487 +0000 UTC m=+410.404121240"
Sep 11 21:59:17.230725 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:59:17.230692    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-m4qsq\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"
Sep 11 21:59:17.233033 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:59:17.233010    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-dxrrp\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="b0c4d72a-eb03-4b1a-9709-3351d7827036" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dxrrp"
Sep 11 21:59:17.235429 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:59:17.235405    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-zj46d\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="1419f007-66b6-44f4-a174-66271f6b3f44" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-zj46d"
Sep 11 21:59:17.237909 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:59:17.237870    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-m4qsq\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"
Sep 11 21:59:17.257308 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.257265    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vjrz7" podStartSLOduration=1.2572510829999999 podStartE2EDuration="1.257251083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:59:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:59:17.25626557 +0000 UTC m=+410.431341959" watchObservedRunningTime="2026-09-11 21:59:17.257251083 +0000 UTC m=+410.432327453"
Sep 11 21:59:17.259123 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:59:17.259102    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-dxrrp\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="b0c4d72a-eb03-4b1a-9709-3351d7827036" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dxrrp"
Sep 11 21:59:17.330363 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.330272    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1419f007-66b6-44f4-a174-66271f6b3f44-extensions-socket-volume\") pod \"1419f007-66b6-44f4-a174-66271f6b3f44\" (UID: \"1419f007-66b6-44f4-a174-66271f6b3f44\") "
Sep 11 21:59:17.330363 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.330338    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7lvf\" (UniqueName: \"kubernetes.io/projected/1419f007-66b6-44f4-a174-66271f6b3f44-kube-api-access-f7lvf\") pod \"1419f007-66b6-44f4-a174-66271f6b3f44\" (UID: \"1419f007-66b6-44f4-a174-66271f6b3f44\") "
Sep 11 21:59:17.330597 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.330571    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1419f007-66b6-44f4-a174-66271f6b3f44-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1419f007-66b6-44f4-a174-66271f6b3f44" (UID: "1419f007-66b6-44f4-a174-66271f6b3f44"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:59:17.332707 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.332665    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1419f007-66b6-44f4-a174-66271f6b3f44-kube-api-access-f7lvf" (OuterVolumeSpecName: "kube-api-access-f7lvf") pod "1419f007-66b6-44f4-a174-66271f6b3f44" (UID: "1419f007-66b6-44f4-a174-66271f6b3f44"). InnerVolumeSpecName "kube-api-access-f7lvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:59:17.334754 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:59:17.334713    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-dxrrp\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="b0c4d72a-eb03-4b1a-9709-3351d7827036" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dxrrp"
Sep 11 21:59:17.336931 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:59:17.336904    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-zj46d\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="1419f007-66b6-44f4-a174-66271f6b3f44" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-zj46d"
Sep 11 21:59:17.337507 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.337481    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1419f007-66b6-44f4-a174-66271f6b3f44" path="/var/lib/kubelet/pods/1419f007-66b6-44f4-a174-66271f6b3f44/volumes"
Sep 11 21:59:17.337770 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.337756    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e" path="/var/lib/kubelet/pods/4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e/volumes"
Sep 11 21:59:17.338081 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.338063    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0c4d72a-eb03-4b1a-9709-3351d7827036" path="/var/lib/kubelet/pods/b0c4d72a-eb03-4b1a-9709-3351d7827036/volumes"
Sep 11 21:59:17.339069 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:59:17.339045    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-m4qsq\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="4b8fc9db-b58d-4fec-9c2f-d68d92a38c9e" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-m4qsq"
Sep 11 21:59:17.431118 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.431088    2587 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1419f007-66b6-44f4-a174-66271f6b3f44-extensions-socket-volume\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:59:17.431118 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:17.431115    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7lvf\" (UniqueName: \"kubernetes.io/projected/1419f007-66b6-44f4-a174-66271f6b3f44-kube-api-access-f7lvf\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:59:18.140055 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:18.140024    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-mcqjh"
Sep 11 21:59:18.214649 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:18.214619    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-zj46d"
Sep 11 21:59:18.215099 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:18.214752    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vjrz7"
Sep 11 21:59:18.219282 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:59:18.219253    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-zj46d\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="1419f007-66b6-44f4-a174-66271f6b3f44" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-zj46d"
Sep 11 21:59:23.139566 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:23.139517    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74996bdd8f-x8pfn" podUID="365e94d9-4eba-4b0f-a851-8aaa665e2ed4" containerName="console" containerID="cri-o://b13334191a1adc1d41b89eee62b014f9b019f84bfee9ce19b44d39ff61391464" gracePeriod=15
Sep 11 21:59:23.390192 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:23.390136    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:59:23.483171 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:23.483129    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-console-serving-cert\") pod \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") "
Sep 11 21:59:23.483171 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:23.483174    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-service-ca\") pod \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") "
Sep 11 21:59:23.483420 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:23.483222    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-trusted-ca-bundle\") pod \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") "
Sep 11 21:59:23.483420 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:23.483256    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-console-oauth-config\") pod \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") "
Sep 11 21:59:23.483420 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:23.483287    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2f7z\" (UniqueName: \"kubernetes.io/projected/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-kube-api-access-l2f7z\") pod \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") "
Sep 11 21:59:23.483420 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:23.483311    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-oauth-serving-cert\") pod \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") "
Sep 11 21:59:23.483420 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:23.483366    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-console-config\") pod \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\" (UID: \"365e94d9-4eba-4b0f-a851-8aaa665e2ed4\") "
Sep 11 21:59:23.483747 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:23.483713    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "365e94d9-4eba-4b0f-a851-8aaa665e2ed4" (UID: "365e94d9-4eba-4b0f-a851-8aaa665e2ed4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:59:23.483837 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:23.483719    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-service-ca" (OuterVolumeSpecName: "service-ca") pod "365e94d9-4eba-4b0f-a851-8aaa665e2ed4" (UID: "365e94d9-4eba-4b0f-a851-8aaa665e2ed4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:59:23.483897 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:23.483863    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "365e94d9-4eba-4b0f-a851-8aaa665e2ed4" (UID: "365e94d9-4eba-4b0f-a851-8aaa665e2ed4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:59:23.483954 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:23.483931    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-console-config" (OuterVolumeSpecName: "console-config") pod "365e94d9-4eba-4b0f-a851-8aaa665e2ed4" (UID: "365e94d9-4eba-4b0f-a851-8aaa665e2ed4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 21:59:23.485542 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:23.485517    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "365e94d9-4eba-4b0f-a851-8aaa665e2ed4" (UID: "365e94d9-4eba-4b0f-a851-8aaa665e2ed4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:59:23.485648 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:23.485596    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-kube-api-access-l2f7z" (OuterVolumeSpecName: "kube-api-access-l2f7z") pod "365e94d9-4eba-4b0f-a851-8aaa665e2ed4" (UID: "365e94d9-4eba-4b0f-a851-8aaa665e2ed4"). InnerVolumeSpecName "kube-api-access-l2f7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:59:23.485648 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:23.485601    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "365e94d9-4eba-4b0f-a851-8aaa665e2ed4" (UID: "365e94d9-4eba-4b0f-a851-8aaa665e2ed4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 21:59:23.584798 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:23.584764    2587 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-console-config\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:59:23.584798 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:23.584793    2587 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-console-serving-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:59:23.584798 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:23.584802    2587 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-service-ca\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:59:23.585010 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:23.584810    2587 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-trusted-ca-bundle\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:59:23.585010 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:23.584820    2587 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-console-oauth-config\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:59:23.585010 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:23.584829    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2f7z\" (UniqueName: \"kubernetes.io/projected/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-kube-api-access-l2f7z\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:59:23.585010 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:23.584837    2587 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/365e94d9-4eba-4b0f-a851-8aaa665e2ed4-oauth-serving-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:59:24.244806 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:24.244772    2587 generic.go:356] "Generic (PLEG): container finished" podID="365e94d9-4eba-4b0f-a851-8aaa665e2ed4" containerID="b13334191a1adc1d41b89eee62b014f9b019f84bfee9ce19b44d39ff61391464" exitCode=0
Sep 11 21:59:24.245269 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:24.244816    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74996bdd8f-x8pfn"
Sep 11 21:59:24.245269 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:24.244861    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74996bdd8f-x8pfn" event={"ID":"365e94d9-4eba-4b0f-a851-8aaa665e2ed4","Type":"ContainerDied","Data":"b13334191a1adc1d41b89eee62b014f9b019f84bfee9ce19b44d39ff61391464"}
Sep 11 21:59:24.245269 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:24.244925    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74996bdd8f-x8pfn" event={"ID":"365e94d9-4eba-4b0f-a851-8aaa665e2ed4","Type":"ContainerDied","Data":"f58a439803bd048b4bb4dba7d08db3700c67af4f3371ae3f999d31be50296458"}
Sep 11 21:59:24.245269 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:24.244958    2587 scope.go:117] "RemoveContainer" containerID="b13334191a1adc1d41b89eee62b014f9b019f84bfee9ce19b44d39ff61391464"
Sep 11 21:59:24.254232 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:24.254215    2587 scope.go:117] "RemoveContainer" containerID="b13334191a1adc1d41b89eee62b014f9b019f84bfee9ce19b44d39ff61391464"
Sep 11 21:59:24.254543 ip-10-0-139-88 kubenswrapper[2587]: E0911 21:59:24.254525    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b13334191a1adc1d41b89eee62b014f9b019f84bfee9ce19b44d39ff61391464\": container with ID starting with b13334191a1adc1d41b89eee62b014f9b019f84bfee9ce19b44d39ff61391464 not found: ID does not exist" containerID="b13334191a1adc1d41b89eee62b014f9b019f84bfee9ce19b44d39ff61391464"
Sep 11 21:59:24.254622 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:24.254551    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b13334191a1adc1d41b89eee62b014f9b019f84bfee9ce19b44d39ff61391464"} err="failed to get container status \"b13334191a1adc1d41b89eee62b014f9b019f84bfee9ce19b44d39ff61391464\": rpc error: code = NotFound desc = could not find container \"b13334191a1adc1d41b89eee62b014f9b019f84bfee9ce19b44d39ff61391464\": container with ID starting with b13334191a1adc1d41b89eee62b014f9b019f84bfee9ce19b44d39ff61391464 not found: ID does not exist"
Sep 11 21:59:24.266489 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:24.266467    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74996bdd8f-x8pfn"]
Sep 11 21:59:24.269635 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:24.269608    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74996bdd8f-x8pfn"]
Sep 11 21:59:25.336202 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:25.336167    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="365e94d9-4eba-4b0f-a851-8aaa665e2ed4" path="/var/lib/kubelet/pods/365e94d9-4eba-4b0f-a851-8aaa665e2ed4/volumes"
Sep 11 21:59:28.217471 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:28.217439    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtff9"
Sep 11 21:59:29.221878 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:29.221844    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-vjrz7"
Sep 11 21:59:33.030647 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:33.030605    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtff9"]
Sep 11 21:59:33.031105 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:33.030836    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtff9" podUID="69f007b1-9538-4d01-9513-d98e88d1f52e" containerName="manager" containerID="cri-o://15b7b79f2862affaade40430762187c51c890f14fd3d4c4a526e6fba495b40a0" gracePeriod=10
Sep 11 21:59:33.289537 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:33.289467    2587 generic.go:356] "Generic (PLEG): container finished" podID="69f007b1-9538-4d01-9513-d98e88d1f52e" containerID="15b7b79f2862affaade40430762187c51c890f14fd3d4c4a526e6fba495b40a0" exitCode=0
Sep 11 21:59:33.289671 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:33.289538    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtff9" event={"ID":"69f007b1-9538-4d01-9513-d98e88d1f52e","Type":"ContainerDied","Data":"15b7b79f2862affaade40430762187c51c890f14fd3d4c4a526e6fba495b40a0"}
Sep 11 21:59:33.800439 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:33.800407    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtff9"
Sep 11 21:59:33.882190 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:33.882153    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mvmv\" (UniqueName: \"kubernetes.io/projected/69f007b1-9538-4d01-9513-d98e88d1f52e-kube-api-access-5mvmv\") pod \"69f007b1-9538-4d01-9513-d98e88d1f52e\" (UID: \"69f007b1-9538-4d01-9513-d98e88d1f52e\") "
Sep 11 21:59:33.882402 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:33.882262    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69f007b1-9538-4d01-9513-d98e88d1f52e-extensions-socket-volume\") pod \"69f007b1-9538-4d01-9513-d98e88d1f52e\" (UID: \"69f007b1-9538-4d01-9513-d98e88d1f52e\") "
Sep 11 21:59:33.882653 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:33.882627    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f007b1-9538-4d01-9513-d98e88d1f52e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "69f007b1-9538-4d01-9513-d98e88d1f52e" (UID: "69f007b1-9538-4d01-9513-d98e88d1f52e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 21:59:33.884319 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:33.884289    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f007b1-9538-4d01-9513-d98e88d1f52e-kube-api-access-5mvmv" (OuterVolumeSpecName: "kube-api-access-5mvmv") pod "69f007b1-9538-4d01-9513-d98e88d1f52e" (UID: "69f007b1-9538-4d01-9513-d98e88d1f52e"). InnerVolumeSpecName "kube-api-access-5mvmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 21:59:33.983029 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:33.982994    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mvmv\" (UniqueName: \"kubernetes.io/projected/69f007b1-9538-4d01-9513-d98e88d1f52e-kube-api-access-5mvmv\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:59:33.983029 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:33.983023    2587 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69f007b1-9538-4d01-9513-d98e88d1f52e-extensions-socket-volume\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 21:59:34.296206 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:34.296132    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtff9"
Sep 11 21:59:34.296206 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:34.296156    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtff9" event={"ID":"69f007b1-9538-4d01-9513-d98e88d1f52e","Type":"ContainerDied","Data":"ac78dd898bdc0e57474bac41161a7815261a4c207ee2889bbb17a2222b7f1839"}
Sep 11 21:59:34.296206 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:34.296208    2587 scope.go:117] "RemoveContainer" containerID="15b7b79f2862affaade40430762187c51c890f14fd3d4c4a526e6fba495b40a0"
Sep 11 21:59:34.318303 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:34.318279    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtff9"]
Sep 11 21:59:34.328784 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:34.328761    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wtff9"]
Sep 11 21:59:35.335162 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:35.335132    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69f007b1-9538-4d01-9513-d98e88d1f52e" path="/var/lib/kubelet/pods/69f007b1-9538-4d01-9513-d98e88d1f52e/volumes"
Sep 11 21:59:49.421021 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.420945    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"]
Sep 11 21:59:49.421429 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.421304    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69f007b1-9538-4d01-9513-d98e88d1f52e" containerName="manager"
Sep 11 21:59:49.421429 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.421317    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="69f007b1-9538-4d01-9513-d98e88d1f52e" containerName="manager"
Sep 11 21:59:49.421429 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.421350    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="365e94d9-4eba-4b0f-a851-8aaa665e2ed4" containerName="console"
Sep 11 21:59:49.421429 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.421356    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="365e94d9-4eba-4b0f-a851-8aaa665e2ed4" containerName="console"
Sep 11 21:59:49.421429 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.421410    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="365e94d9-4eba-4b0f-a851-8aaa665e2ed4" containerName="console"
Sep 11 21:59:49.421429 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.421417    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="69f007b1-9538-4d01-9513-d98e88d1f52e" containerName="manager"
Sep 11 21:59:49.425300 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.425280    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.428219 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.428124    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 21:59:49.428387 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.428231    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2qxmz\""
Sep 11 21:59:49.445008 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.444964    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"]
Sep 11 21:59:49.609156 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.609124    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60203398-7954-425d-8699-a6c2bfd7bddb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.609344 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.609163    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60203398-7954-425d-8699-a6c2bfd7bddb-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.609344 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.609187    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60203398-7954-425d-8699-a6c2bfd7bddb-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.609344 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.609212    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60203398-7954-425d-8699-a6c2bfd7bddb-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.609344 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.609311    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60203398-7954-425d-8699-a6c2bfd7bddb-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.609487 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.609363    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmbnb\" (UniqueName: \"kubernetes.io/projected/60203398-7954-425d-8699-a6c2bfd7bddb-kube-api-access-zmbnb\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.609487 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.609434    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60203398-7954-425d-8699-a6c2bfd7bddb-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.609487 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.609467    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60203398-7954-425d-8699-a6c2bfd7bddb-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.609578 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.609523    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60203398-7954-425d-8699-a6c2bfd7bddb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.710418 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.710315    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60203398-7954-425d-8699-a6c2bfd7bddb-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.710418 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.710389    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60203398-7954-425d-8699-a6c2bfd7bddb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.710418 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.710420    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60203398-7954-425d-8699-a6c2bfd7bddb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.710696 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.710440    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60203398-7954-425d-8699-a6c2bfd7bddb-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.710696 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.710464    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60203398-7954-425d-8699-a6c2bfd7bddb-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.710696 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.710499    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60203398-7954-425d-8699-a6c2bfd7bddb-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.710696 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.710533    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60203398-7954-425d-8699-a6c2bfd7bddb-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.710696 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.710562    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmbnb\" (UniqueName: \"kubernetes.io/projected/60203398-7954-425d-8699-a6c2bfd7bddb-kube-api-access-zmbnb\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.710696 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.710593    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60203398-7954-425d-8699-a6c2bfd7bddb-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.711000 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.710873    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60203398-7954-425d-8699-a6c2bfd7bddb-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.711148 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.711118    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60203398-7954-425d-8699-a6c2bfd7bddb-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.711355 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.711179    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60203398-7954-425d-8699-a6c2bfd7bddb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.711355 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.711191    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60203398-7954-425d-8699-a6c2bfd7bddb-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.711355 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.711310    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60203398-7954-425d-8699-a6c2bfd7bddb-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.712819 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.712792    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60203398-7954-425d-8699-a6c2bfd7bddb-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.712921 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.712905    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60203398-7954-425d-8699-a6c2bfd7bddb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.720409 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.720383    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60203398-7954-425d-8699-a6c2bfd7bddb-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.720553 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.720530    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmbnb\" (UniqueName: \"kubernetes.io/projected/60203398-7954-425d-8699-a6c2bfd7bddb-kube-api-access-zmbnb\") pod \"maas-default-gateway-openshift-default-78bb78ff78-jtsp7\" (UID: \"60203398-7954-425d-8699-a6c2bfd7bddb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:49.736120 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:49.736092    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:50.014769 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:50.014736    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"]
Sep 11 21:59:50.015846 ip-10-0-139-88 kubenswrapper[2587]: W0911 21:59:50.015817    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60203398_7954_425d_8699_a6c2bfd7bddb.slice/crio-885241fa14a04876e2e0cf53aa08e497dafbc25a5bed375c9476df76fb463c47 WatchSource:0}: Error finding container 885241fa14a04876e2e0cf53aa08e497dafbc25a5bed375c9476df76fb463c47: Status 404 returned error can't find the container with id 885241fa14a04876e2e0cf53aa08e497dafbc25a5bed375c9476df76fb463c47
Sep 11 21:59:50.374533 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:50.374439    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7" event={"ID":"60203398-7954-425d-8699-a6c2bfd7bddb","Type":"ContainerStarted","Data":"885241fa14a04876e2e0cf53aa08e497dafbc25a5bed375c9476df76fb463c47"}
Sep 11 21:59:58.154031 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:58.153994    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 21:59:58.154309 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:58.154068    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 21:59:58.154309 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:58.154096    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 21:59:58.415985 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:58.415942    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7" event={"ID":"60203398-7954-425d-8699-a6c2bfd7bddb","Type":"ContainerStarted","Data":"c1e0a32a7d76604232000ca342abc36e7636bcb7d3f9c3f3f52f2accdcb182d1"}
Sep 11 21:59:58.439072 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:58.439010    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7" podStartSLOduration=1.303156363 podStartE2EDuration="9.438989671s" totalImagesPullingTime="8.135833308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:59:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 21:59:58.437875111 +0000 UTC m=+451.612951503" watchObservedRunningTime="2026-09-11 21:59:58.438989671 +0000 UTC m=+451.614066059"
Sep 11 21:59:58.736526 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:58.736435    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:59.435088 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:59.435043    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r46sk"]
Sep 11 21:59:59.438320 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:59.438302    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-r46sk"
Sep 11 21:59:59.440955 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:59.440936    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 21:59:59.446486 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:59.446469    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r46sk"]
Sep 11 21:59:59.498352 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:59.498306    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a0c1bc93-f829-4740-9409-7f7ee38ecd70-config-file\") pod \"limitador-limitador-8588749977-r46sk\" (UID: \"a0c1bc93-f829-4740-9409-7f7ee38ecd70\") " pod="kuadrant-system/limitador-limitador-8588749977-r46sk"
Sep 11 21:59:59.498532 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:59.498413    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2j49\" (UniqueName: \"kubernetes.io/projected/a0c1bc93-f829-4740-9409-7f7ee38ecd70-kube-api-access-n2j49\") pod \"limitador-limitador-8588749977-r46sk\" (UID: \"a0c1bc93-f829-4740-9409-7f7ee38ecd70\") " pod="kuadrant-system/limitador-limitador-8588749977-r46sk"
Sep 11 21:59:59.531469 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:59.531435    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r46sk"]
Sep 11 21:59:59.599468 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:59.599427    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2j49\" (UniqueName: \"kubernetes.io/projected/a0c1bc93-f829-4740-9409-7f7ee38ecd70-kube-api-access-n2j49\") pod \"limitador-limitador-8588749977-r46sk\" (UID: \"a0c1bc93-f829-4740-9409-7f7ee38ecd70\") " pod="kuadrant-system/limitador-limitador-8588749977-r46sk"
Sep 11 21:59:59.599679 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:59.599504    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a0c1bc93-f829-4740-9409-7f7ee38ecd70-config-file\") pod \"limitador-limitador-8588749977-r46sk\" (UID: \"a0c1bc93-f829-4740-9409-7f7ee38ecd70\") " pod="kuadrant-system/limitador-limitador-8588749977-r46sk"
Sep 11 21:59:59.600097 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:59.600075    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a0c1bc93-f829-4740-9409-7f7ee38ecd70-config-file\") pod \"limitador-limitador-8588749977-r46sk\" (UID: \"a0c1bc93-f829-4740-9409-7f7ee38ecd70\") " pod="kuadrant-system/limitador-limitador-8588749977-r46sk"
Sep 11 21:59:59.608372 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:59.608313    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2j49\" (UniqueName: \"kubernetes.io/projected/a0c1bc93-f829-4740-9409-7f7ee38ecd70-kube-api-access-n2j49\") pod \"limitador-limitador-8588749977-r46sk\" (UID: \"a0c1bc93-f829-4740-9409-7f7ee38ecd70\") " pod="kuadrant-system/limitador-limitador-8588749977-r46sk"
Sep 11 21:59:59.741922 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:59.741840    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 21:59:59.748601 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:59.748555    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-r46sk"
Sep 11 21:59:59.896354 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:59.896309    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 21:59:59.900728 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:59.900706    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"
Sep 11 21:59:59.908066 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:59.908037    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 21:59:59.939260 ip-10-0-139-88 kubenswrapper[2587]: I0911 21:59:59.939225    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:00:00.003920 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:00.003838    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jws7n\" (UniqueName: \"kubernetes.io/projected/13759088-b5e2-4daa-8795-496e47f511eb-kube-api-access-jws7n\") pod \"limitador-limitador-84994f7ffc-tw9k5\" (UID: \"13759088-b5e2-4daa-8795-496e47f511eb\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"
Sep 11 22:00:00.003920 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:00.003883    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/13759088-b5e2-4daa-8795-496e47f511eb-config-file\") pod \"limitador-limitador-84994f7ffc-tw9k5\" (UID: \"13759088-b5e2-4daa-8795-496e47f511eb\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"
Sep 11 22:00:00.018864 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:00.018838    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r46sk"]
Sep 11 22:00:00.020844 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:00:00.020817    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c1bc93_f829_4740_9409_7f7ee38ecd70.slice/crio-bcc6d716193935f5a7c77272e469bd23e89335cb42004afa02f5f4f6460cb05d WatchSource:0}: Error finding container bcc6d716193935f5a7c77272e469bd23e89335cb42004afa02f5f4f6460cb05d: Status 404 returned error can't find the container with id bcc6d716193935f5a7c77272e469bd23e89335cb42004afa02f5f4f6460cb05d
Sep 11 22:00:00.104655 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:00.104621    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jws7n\" (UniqueName: \"kubernetes.io/projected/13759088-b5e2-4daa-8795-496e47f511eb-kube-api-access-jws7n\") pod \"limitador-limitador-84994f7ffc-tw9k5\" (UID: \"13759088-b5e2-4daa-8795-496e47f511eb\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"
Sep 11 22:00:00.104818 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:00.104670    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/13759088-b5e2-4daa-8795-496e47f511eb-config-file\") pod \"limitador-limitador-84994f7ffc-tw9k5\" (UID: \"13759088-b5e2-4daa-8795-496e47f511eb\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"
Sep 11 22:00:00.105226 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:00.105209    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/13759088-b5e2-4daa-8795-496e47f511eb-config-file\") pod \"limitador-limitador-84994f7ffc-tw9k5\" (UID: \"13759088-b5e2-4daa-8795-496e47f511eb\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"
Sep 11 22:00:00.113112 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:00.113090    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jws7n\" (UniqueName: \"kubernetes.io/projected/13759088-b5e2-4daa-8795-496e47f511eb-kube-api-access-jws7n\") pod \"limitador-limitador-84994f7ffc-tw9k5\" (UID: \"13759088-b5e2-4daa-8795-496e47f511eb\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"
Sep 11 22:00:00.212394 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:00.212355    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"
Sep 11 22:00:00.426219 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:00.426181    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-r46sk" event={"ID":"a0c1bc93-f829-4740-9409-7f7ee38ecd70","Type":"ContainerStarted","Data":"bcc6d716193935f5a7c77272e469bd23e89335cb42004afa02f5f4f6460cb05d"}
Sep 11 22:00:00.426438 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:00.426424    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 22:00:00.427446 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:00.427425    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 22:00:00.484928 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:00.484861    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:00:00.487591 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:00:00.487563    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13759088_b5e2_4daa_8795_496e47f511eb.slice/crio-b5b39aee27a9d333035b36c6e41efcc03b535636442b1f45769b49e297190087 WatchSource:0}: Error finding container b5b39aee27a9d333035b36c6e41efcc03b535636442b1f45769b49e297190087: Status 404 returned error can't find the container with id b5b39aee27a9d333035b36c6e41efcc03b535636442b1f45769b49e297190087
Sep 11 22:00:01.432587 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:01.432466    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-tw9k5" event={"ID":"13759088-b5e2-4daa-8795-496e47f511eb","Type":"ContainerStarted","Data":"b5b39aee27a9d333035b36c6e41efcc03b535636442b1f45769b49e297190087"}
Sep 11 22:00:02.438861 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:02.438826    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-tw9k5" event={"ID":"13759088-b5e2-4daa-8795-496e47f511eb","Type":"ContainerStarted","Data":"51fc8036ead9bcce7da2f2965d4774851b915d535b26bf8637e5d5e30fb2f3a7"}
Sep 11 22:00:02.439347 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:02.438899    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"
Sep 11 22:00:02.440520 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:02.440495    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-r46sk" event={"ID":"a0c1bc93-f829-4740-9409-7f7ee38ecd70","Type":"ContainerStarted","Data":"690dec913959f52390739fcaa363221fbf4ceab4857fc1f166e2c2e7fc5f696c"}
Sep 11 22:00:02.440635 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:02.440543    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-r46sk"
Sep 11 22:00:02.458506 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:02.458468    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-tw9k5" podStartSLOduration=2.518336869 podStartE2EDuration="3.458454147s" totalImagesPullingTime="940.117278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:59:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:00:02.456391724 +0000 UTC m=+455.631468119" watchObservedRunningTime="2026-09-11 22:00:02.458454147 +0000 UTC m=+455.633530524"
Sep 11 22:00:02.473790 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:02.473749    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-r46sk" podStartSLOduration=2.065109794 podStartE2EDuration="3.473735331s" totalImagesPullingTime="1.408625537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 21:59:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:00:02.472408 +0000 UTC m=+455.647484392" watchObservedRunningTime="2026-09-11 22:00:02.473735331 +0000 UTC m=+455.648811754"
Sep 11 22:00:13.446307 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:13.446268    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"
Sep 11 22:00:13.446750 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:13.446373    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-r46sk"
Sep 11 22:00:13.514649 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:13.514615    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r46sk"]
Sep 11 22:00:13.514852 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:13.514825    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-r46sk" podUID="a0c1bc93-f829-4740-9409-7f7ee38ecd70" containerName="limitador" containerID="cri-o://690dec913959f52390739fcaa363221fbf4ceab4857fc1f166e2c2e7fc5f696c" gracePeriod=30
Sep 11 22:00:14.049674 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:14.049647    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-r46sk"
Sep 11 22:00:14.129827 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:14.129733    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2j49\" (UniqueName: \"kubernetes.io/projected/a0c1bc93-f829-4740-9409-7f7ee38ecd70-kube-api-access-n2j49\") pod \"a0c1bc93-f829-4740-9409-7f7ee38ecd70\" (UID: \"a0c1bc93-f829-4740-9409-7f7ee38ecd70\") "
Sep 11 22:00:14.129827 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:14.129785    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a0c1bc93-f829-4740-9409-7f7ee38ecd70-config-file\") pod \"a0c1bc93-f829-4740-9409-7f7ee38ecd70\" (UID: \"a0c1bc93-f829-4740-9409-7f7ee38ecd70\") "
Sep 11 22:00:14.130158 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:14.130132    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c1bc93-f829-4740-9409-7f7ee38ecd70-config-file" (OuterVolumeSpecName: "config-file") pod "a0c1bc93-f829-4740-9409-7f7ee38ecd70" (UID: "a0c1bc93-f829-4740-9409-7f7ee38ecd70"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:00:14.131905 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:14.131880    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c1bc93-f829-4740-9409-7f7ee38ecd70-kube-api-access-n2j49" (OuterVolumeSpecName: "kube-api-access-n2j49") pod "a0c1bc93-f829-4740-9409-7f7ee38ecd70" (UID: "a0c1bc93-f829-4740-9409-7f7ee38ecd70"). InnerVolumeSpecName "kube-api-access-n2j49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:00:14.230636 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:14.230594    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2j49\" (UniqueName: \"kubernetes.io/projected/a0c1bc93-f829-4740-9409-7f7ee38ecd70-kube-api-access-n2j49\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:00:14.230636 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:14.230626    2587 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a0c1bc93-f829-4740-9409-7f7ee38ecd70-config-file\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:00:14.502625 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:14.502590    2587 generic.go:356] "Generic (PLEG): container finished" podID="a0c1bc93-f829-4740-9409-7f7ee38ecd70" containerID="690dec913959f52390739fcaa363221fbf4ceab4857fc1f166e2c2e7fc5f696c" exitCode=0
Sep 11 22:00:14.502994 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:14.502625    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-r46sk" event={"ID":"a0c1bc93-f829-4740-9409-7f7ee38ecd70","Type":"ContainerDied","Data":"690dec913959f52390739fcaa363221fbf4ceab4857fc1f166e2c2e7fc5f696c"}
Sep 11 22:00:14.502994 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:14.502666    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-r46sk"
Sep 11 22:00:14.502994 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:14.502684    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-r46sk" event={"ID":"a0c1bc93-f829-4740-9409-7f7ee38ecd70","Type":"ContainerDied","Data":"bcc6d716193935f5a7c77272e469bd23e89335cb42004afa02f5f4f6460cb05d"}
Sep 11 22:00:14.502994 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:14.502709    2587 scope.go:117] "RemoveContainer" containerID="690dec913959f52390739fcaa363221fbf4ceab4857fc1f166e2c2e7fc5f696c"
Sep 11 22:00:14.511135 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:14.511107    2587 scope.go:117] "RemoveContainer" containerID="690dec913959f52390739fcaa363221fbf4ceab4857fc1f166e2c2e7fc5f696c"
Sep 11 22:00:14.511426 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:00:14.511401    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"690dec913959f52390739fcaa363221fbf4ceab4857fc1f166e2c2e7fc5f696c\": container with ID starting with 690dec913959f52390739fcaa363221fbf4ceab4857fc1f166e2c2e7fc5f696c not found: ID does not exist" containerID="690dec913959f52390739fcaa363221fbf4ceab4857fc1f166e2c2e7fc5f696c"
Sep 11 22:00:14.511506 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:14.511433    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690dec913959f52390739fcaa363221fbf4ceab4857fc1f166e2c2e7fc5f696c"} err="failed to get container status \"690dec913959f52390739fcaa363221fbf4ceab4857fc1f166e2c2e7fc5f696c\": rpc error: code = NotFound desc = could not find container \"690dec913959f52390739fcaa363221fbf4ceab4857fc1f166e2c2e7fc5f696c\": container with ID starting with 690dec913959f52390739fcaa363221fbf4ceab4857fc1f166e2c2e7fc5f696c not found: ID does not exist"
Sep 11 22:00:14.525603 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:14.525569    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r46sk"]
Sep 11 22:00:14.529663 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:14.529639    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-r46sk"]
Sep 11 22:00:15.335823 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:15.335790    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0c1bc93-f829-4740-9409-7f7ee38ecd70" path="/var/lib/kubelet/pods/a0c1bc93-f829-4740-9409-7f7ee38ecd70/volumes"
Sep 11 22:00:31.167724 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:31.167685    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-f55d54cdf-26nvb"]
Sep 11 22:00:31.168282 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:31.168202    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0c1bc93-f829-4740-9409-7f7ee38ecd70" containerName="limitador"
Sep 11 22:00:31.168282 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:31.168224    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0c1bc93-f829-4740-9409-7f7ee38ecd70" containerName="limitador"
Sep 11 22:00:31.168425 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:31.168310    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0c1bc93-f829-4740-9409-7f7ee38ecd70" containerName="limitador"
Sep 11 22:00:31.172804 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:31.172782    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f55d54cdf-26nvb"
Sep 11 22:00:31.175577 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:31.175548    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-74fxz\""
Sep 11 22:00:31.177108 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:31.177080    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 11 22:00:31.179811 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:31.179788    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f55d54cdf-26nvb"]
Sep 11 22:00:31.285942 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:31.285888    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3264d17b-c9fd-4959-a492-a368e7c17249-tls-cert\") pod \"authorino-f55d54cdf-26nvb\" (UID: \"3264d17b-c9fd-4959-a492-a368e7c17249\") " pod="kuadrant-system/authorino-f55d54cdf-26nvb"
Sep 11 22:00:31.286146 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:31.286045    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-774bp\" (UniqueName: \"kubernetes.io/projected/3264d17b-c9fd-4959-a492-a368e7c17249-kube-api-access-774bp\") pod \"authorino-f55d54cdf-26nvb\" (UID: \"3264d17b-c9fd-4959-a492-a368e7c17249\") " pod="kuadrant-system/authorino-f55d54cdf-26nvb"
Sep 11 22:00:31.386705 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:31.386668    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-774bp\" (UniqueName: \"kubernetes.io/projected/3264d17b-c9fd-4959-a492-a368e7c17249-kube-api-access-774bp\") pod \"authorino-f55d54cdf-26nvb\" (UID: \"3264d17b-c9fd-4959-a492-a368e7c17249\") " pod="kuadrant-system/authorino-f55d54cdf-26nvb"
Sep 11 22:00:31.386705 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:31.386726    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3264d17b-c9fd-4959-a492-a368e7c17249-tls-cert\") pod \"authorino-f55d54cdf-26nvb\" (UID: \"3264d17b-c9fd-4959-a492-a368e7c17249\") " pod="kuadrant-system/authorino-f55d54cdf-26nvb"
Sep 11 22:00:31.389247 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:31.389218    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3264d17b-c9fd-4959-a492-a368e7c17249-tls-cert\") pod \"authorino-f55d54cdf-26nvb\" (UID: \"3264d17b-c9fd-4959-a492-a368e7c17249\") " pod="kuadrant-system/authorino-f55d54cdf-26nvb"
Sep 11 22:00:31.396869 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:31.396840    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-774bp\" (UniqueName: \"kubernetes.io/projected/3264d17b-c9fd-4959-a492-a368e7c17249-kube-api-access-774bp\") pod \"authorino-f55d54cdf-26nvb\" (UID: \"3264d17b-c9fd-4959-a492-a368e7c17249\") " pod="kuadrant-system/authorino-f55d54cdf-26nvb"
Sep 11 22:00:31.483733 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:31.483654    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f55d54cdf-26nvb"
Sep 11 22:00:31.753612 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:31.753531    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f55d54cdf-26nvb"]
Sep 11 22:00:31.756811 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:00:31.756780    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3264d17b_c9fd_4959_a492_a368e7c17249.slice/crio-d916db52be59169634118d31f61cebc9ef690529970bc5dc49b442959d979d0b WatchSource:0}: Error finding container d916db52be59169634118d31f61cebc9ef690529970bc5dc49b442959d979d0b: Status 404 returned error can't find the container with id d916db52be59169634118d31f61cebc9ef690529970bc5dc49b442959d979d0b
Sep 11 22:00:32.590498 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:32.590464    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f55d54cdf-26nvb" event={"ID":"3264d17b-c9fd-4959-a492-a368e7c17249","Type":"ContainerStarted","Data":"d916db52be59169634118d31f61cebc9ef690529970bc5dc49b442959d979d0b"}
Sep 11 22:00:34.605102 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:34.605062    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f55d54cdf-26nvb" event={"ID":"3264d17b-c9fd-4959-a492-a368e7c17249","Type":"ContainerStarted","Data":"a8d3cade4c6e9a7a748c4c2961e153dd0510d8452dd0472da22d3c6035fef9a2"}
Sep 11 22:00:34.623381 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:34.623309    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-f55d54cdf-26nvb" podStartSLOduration=1.598189896 podStartE2EDuration="3.623289984s" totalImagesPullingTime="2.025100088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:00:34.622627351 +0000 UTC m=+487.797703721" watchObservedRunningTime="2026-09-11 22:00:34.623289984 +0000 UTC m=+487.798366353"
Sep 11 22:00:39.506534 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.506499    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5f4db57c75-fnkjr"]
Sep 11 22:00:39.510397 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.510370    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f4db57c75-fnkjr"
Sep 11 22:00:39.515048 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.515021    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 22:00:39.516299 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.516276    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 22:00:39.516433 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.516288    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 22:00:39.516433 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.516278    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kwx56\""
Sep 11 22:00:39.538309 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.538276    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f4db57c75-fnkjr"]
Sep 11 22:00:39.557387 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.557350    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2q68\" (UniqueName: \"kubernetes.io/projected/36dc7329-32af-457b-aa1e-e0a38d67292a-kube-api-access-r2q68\") pod \"maas-controller-5f4db57c75-fnkjr\" (UID: \"36dc7329-32af-457b-aa1e-e0a38d67292a\") " pod="opendatahub/maas-controller-5f4db57c75-fnkjr"
Sep 11 22:00:39.557565 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.557420    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36dc7329-32af-457b-aa1e-e0a38d67292a-webhook-cert\") pod \"maas-controller-5f4db57c75-fnkjr\" (UID: \"36dc7329-32af-457b-aa1e-e0a38d67292a\") " pod="opendatahub/maas-controller-5f4db57c75-fnkjr"
Sep 11 22:00:39.557565 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.557468    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36dc7329-32af-457b-aa1e-e0a38d67292a-metrics-certs\") pod \"maas-controller-5f4db57c75-fnkjr\" (UID: \"36dc7329-32af-457b-aa1e-e0a38d67292a\") " pod="opendatahub/maas-controller-5f4db57c75-fnkjr"
Sep 11 22:00:39.658755 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.658719    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2q68\" (UniqueName: \"kubernetes.io/projected/36dc7329-32af-457b-aa1e-e0a38d67292a-kube-api-access-r2q68\") pod \"maas-controller-5f4db57c75-fnkjr\" (UID: \"36dc7329-32af-457b-aa1e-e0a38d67292a\") " pod="opendatahub/maas-controller-5f4db57c75-fnkjr"
Sep 11 22:00:39.658934 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.658764    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36dc7329-32af-457b-aa1e-e0a38d67292a-webhook-cert\") pod \"maas-controller-5f4db57c75-fnkjr\" (UID: \"36dc7329-32af-457b-aa1e-e0a38d67292a\") " pod="opendatahub/maas-controller-5f4db57c75-fnkjr"
Sep 11 22:00:39.658934 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.658875    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36dc7329-32af-457b-aa1e-e0a38d67292a-metrics-certs\") pod \"maas-controller-5f4db57c75-fnkjr\" (UID: \"36dc7329-32af-457b-aa1e-e0a38d67292a\") " pod="opendatahub/maas-controller-5f4db57c75-fnkjr"
Sep 11 22:00:39.661419 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.661395    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36dc7329-32af-457b-aa1e-e0a38d67292a-metrics-certs\") pod \"maas-controller-5f4db57c75-fnkjr\" (UID: \"36dc7329-32af-457b-aa1e-e0a38d67292a\") " pod="opendatahub/maas-controller-5f4db57c75-fnkjr"
Sep 11 22:00:39.661531 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.661395    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36dc7329-32af-457b-aa1e-e0a38d67292a-webhook-cert\") pod \"maas-controller-5f4db57c75-fnkjr\" (UID: \"36dc7329-32af-457b-aa1e-e0a38d67292a\") " pod="opendatahub/maas-controller-5f4db57c75-fnkjr"
Sep 11 22:00:39.670727 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.670704    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2q68\" (UniqueName: \"kubernetes.io/projected/36dc7329-32af-457b-aa1e-e0a38d67292a-kube-api-access-r2q68\") pod \"maas-controller-5f4db57c75-fnkjr\" (UID: \"36dc7329-32af-457b-aa1e-e0a38d67292a\") " pod="opendatahub/maas-controller-5f4db57c75-fnkjr"
Sep 11 22:00:39.710111 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.710077    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fcd74f577-d2b6p"]
Sep 11 22:00:39.713525 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.713509    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fcd74f577-d2b6p"
Sep 11 22:00:39.723980 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.723956    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fcd74f577-d2b6p"]
Sep 11 22:00:39.759570 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.759484    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/828b0b6b-f551-4582-8920-0ea68e295a50-webhook-cert\") pod \"maas-controller-5fcd74f577-d2b6p\" (UID: \"828b0b6b-f551-4582-8920-0ea68e295a50\") " pod="opendatahub/maas-controller-5fcd74f577-d2b6p"
Sep 11 22:00:39.759570 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.759528    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52kjz\" (UniqueName: \"kubernetes.io/projected/828b0b6b-f551-4582-8920-0ea68e295a50-kube-api-access-52kjz\") pod \"maas-controller-5fcd74f577-d2b6p\" (UID: \"828b0b6b-f551-4582-8920-0ea68e295a50\") " pod="opendatahub/maas-controller-5fcd74f577-d2b6p"
Sep 11 22:00:39.759744 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.759583    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/828b0b6b-f551-4582-8920-0ea68e295a50-metrics-certs\") pod \"maas-controller-5fcd74f577-d2b6p\" (UID: \"828b0b6b-f551-4582-8920-0ea68e295a50\") " pod="opendatahub/maas-controller-5fcd74f577-d2b6p"
Sep 11 22:00:39.820587 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.820547    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f4db57c75-fnkjr"
Sep 11 22:00:39.860892 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.860856    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52kjz\" (UniqueName: \"kubernetes.io/projected/828b0b6b-f551-4582-8920-0ea68e295a50-kube-api-access-52kjz\") pod \"maas-controller-5fcd74f577-d2b6p\" (UID: \"828b0b6b-f551-4582-8920-0ea68e295a50\") " pod="opendatahub/maas-controller-5fcd74f577-d2b6p"
Sep 11 22:00:39.861074 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.860928    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/828b0b6b-f551-4582-8920-0ea68e295a50-metrics-certs\") pod \"maas-controller-5fcd74f577-d2b6p\" (UID: \"828b0b6b-f551-4582-8920-0ea68e295a50\") " pod="opendatahub/maas-controller-5fcd74f577-d2b6p"
Sep 11 22:00:39.861074 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.861005    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/828b0b6b-f551-4582-8920-0ea68e295a50-webhook-cert\") pod \"maas-controller-5fcd74f577-d2b6p\" (UID: \"828b0b6b-f551-4582-8920-0ea68e295a50\") " pod="opendatahub/maas-controller-5fcd74f577-d2b6p"
Sep 11 22:00:39.863443 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.863416    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/828b0b6b-f551-4582-8920-0ea68e295a50-webhook-cert\") pod \"maas-controller-5fcd74f577-d2b6p\" (UID: \"828b0b6b-f551-4582-8920-0ea68e295a50\") " pod="opendatahub/maas-controller-5fcd74f577-d2b6p"
Sep 11 22:00:39.863582 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.863562    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/828b0b6b-f551-4582-8920-0ea68e295a50-metrics-certs\") pod \"maas-controller-5fcd74f577-d2b6p\" (UID: \"828b0b6b-f551-4582-8920-0ea68e295a50\") " pod="opendatahub/maas-controller-5fcd74f577-d2b6p"
Sep 11 22:00:39.870219 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:39.870184    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52kjz\" (UniqueName: \"kubernetes.io/projected/828b0b6b-f551-4582-8920-0ea68e295a50-kube-api-access-52kjz\") pod \"maas-controller-5fcd74f577-d2b6p\" (UID: \"828b0b6b-f551-4582-8920-0ea68e295a50\") " pod="opendatahub/maas-controller-5fcd74f577-d2b6p"
Sep 11 22:00:40.025095 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:40.025015    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fcd74f577-d2b6p"
Sep 11 22:00:40.094098 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:40.094063    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f4db57c75-fnkjr"]
Sep 11 22:00:40.095046 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:00:40.095020    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36dc7329_32af_457b_aa1e_e0a38d67292a.slice/crio-049c9b8f09eb88f088982351c65cb415d4b584bdc3cd156c77e21acc3552b47e WatchSource:0}: Error finding container 049c9b8f09eb88f088982351c65cb415d4b584bdc3cd156c77e21acc3552b47e: Status 404 returned error can't find the container with id 049c9b8f09eb88f088982351c65cb415d4b584bdc3cd156c77e21acc3552b47e
Sep 11 22:00:40.296696 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:40.296672    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fcd74f577-d2b6p"]
Sep 11 22:00:40.298367 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:00:40.298337    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod828b0b6b_f551_4582_8920_0ea68e295a50.slice/crio-ee5b176e355e9e1b8ef4638cc57fe6d209347464e4ee2c732e4a94e17272b1b7 WatchSource:0}: Error finding container ee5b176e355e9e1b8ef4638cc57fe6d209347464e4ee2c732e4a94e17272b1b7: Status 404 returned error can't find the container with id ee5b176e355e9e1b8ef4638cc57fe6d209347464e4ee2c732e4a94e17272b1b7
Sep 11 22:00:40.633672 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:40.633579    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f4db57c75-fnkjr" event={"ID":"36dc7329-32af-457b-aa1e-e0a38d67292a","Type":"ContainerStarted","Data":"049c9b8f09eb88f088982351c65cb415d4b584bdc3cd156c77e21acc3552b47e"}
Sep 11 22:00:40.634752 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:40.634724    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fcd74f577-d2b6p" event={"ID":"828b0b6b-f551-4582-8920-0ea68e295a50","Type":"ContainerStarted","Data":"ee5b176e355e9e1b8ef4638cc57fe6d209347464e4ee2c732e4a94e17272b1b7"}
Sep 11 22:00:43.652752 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:43.652710    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f4db57c75-fnkjr" event={"ID":"36dc7329-32af-457b-aa1e-e0a38d67292a","Type":"ContainerStarted","Data":"28bcb00d2fcfa8221fdb5eb94fbc24788d01a2d821878cbf52d69276723e1da2"}
Sep 11 22:00:43.653240 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:43.652866    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5f4db57c75-fnkjr"
Sep 11 22:00:43.654250 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:43.654224    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fcd74f577-d2b6p" event={"ID":"828b0b6b-f551-4582-8920-0ea68e295a50","Type":"ContainerStarted","Data":"32a865fe74dc645d77e06e62f870d89aaa070fe6b312505dcc35926bfbbb2564"}
Sep 11 22:00:43.654393 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:43.654265    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fcd74f577-d2b6p"
Sep 11 22:00:43.674764 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:43.674714    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5f4db57c75-fnkjr" podStartSLOduration=1.9471882790000001 podStartE2EDuration="4.674698316s" totalImagesPullingTime="2.727510037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:00:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:00:43.672931651 +0000 UTC m=+496.848008042" watchObservedRunningTime="2026-09-11 22:00:43.674698316 +0000 UTC m=+496.849774686"
Sep 11 22:00:43.693833 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:43.693777    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5fcd74f577-d2b6p" podStartSLOduration=2.162847425 podStartE2EDuration="4.693761153s" totalImagesPullingTime="2.530913728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:00:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:00:43.693060854 +0000 UTC m=+496.868137225" watchObservedRunningTime="2026-09-11 22:00:43.693761153 +0000 UTC m=+496.868837523"
Sep 11 22:00:54.661037 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:54.661006    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fcd74f577-d2b6p"
Sep 11 22:00:54.661413 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:54.661093    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5f4db57c75-fnkjr"
Sep 11 22:00:54.725753 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:54.725720    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f4db57c75-fnkjr"]
Sep 11 22:00:54.725956 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:54.725909    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5f4db57c75-fnkjr" podUID="36dc7329-32af-457b-aa1e-e0a38d67292a" containerName="manager" containerID="cri-o://28bcb00d2fcfa8221fdb5eb94fbc24788d01a2d821878cbf52d69276723e1da2" gracePeriod=30
Sep 11 22:00:54.976815 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:54.976793    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f4db57c75-fnkjr"
Sep 11 22:00:55.100228 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:55.100194    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2q68\" (UniqueName: \"kubernetes.io/projected/36dc7329-32af-457b-aa1e-e0a38d67292a-kube-api-access-r2q68\") pod \"36dc7329-32af-457b-aa1e-e0a38d67292a\" (UID: \"36dc7329-32af-457b-aa1e-e0a38d67292a\") "
Sep 11 22:00:55.100228 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:55.100234    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36dc7329-32af-457b-aa1e-e0a38d67292a-webhook-cert\") pod \"36dc7329-32af-457b-aa1e-e0a38d67292a\" (UID: \"36dc7329-32af-457b-aa1e-e0a38d67292a\") "
Sep 11 22:00:55.100440 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:55.100316    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36dc7329-32af-457b-aa1e-e0a38d67292a-metrics-certs\") pod \"36dc7329-32af-457b-aa1e-e0a38d67292a\" (UID: \"36dc7329-32af-457b-aa1e-e0a38d67292a\") "
Sep 11 22:00:55.102430 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:55.102392    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36dc7329-32af-457b-aa1e-e0a38d67292a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "36dc7329-32af-457b-aa1e-e0a38d67292a" (UID: "36dc7329-32af-457b-aa1e-e0a38d67292a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:00:55.102430 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:55.102408    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36dc7329-32af-457b-aa1e-e0a38d67292a-kube-api-access-r2q68" (OuterVolumeSpecName: "kube-api-access-r2q68") pod "36dc7329-32af-457b-aa1e-e0a38d67292a" (UID: "36dc7329-32af-457b-aa1e-e0a38d67292a"). InnerVolumeSpecName "kube-api-access-r2q68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:00:55.102578 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:55.102430    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36dc7329-32af-457b-aa1e-e0a38d67292a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "36dc7329-32af-457b-aa1e-e0a38d67292a" (UID: "36dc7329-32af-457b-aa1e-e0a38d67292a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:00:55.201285 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:55.201246    2587 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36dc7329-32af-457b-aa1e-e0a38d67292a-metrics-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:00:55.201285 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:55.201276    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2q68\" (UniqueName: \"kubernetes.io/projected/36dc7329-32af-457b-aa1e-e0a38d67292a-kube-api-access-r2q68\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:00:55.201285 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:55.201286    2587 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/36dc7329-32af-457b-aa1e-e0a38d67292a-webhook-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:00:55.713528 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:55.713494    2587 generic.go:356] "Generic (PLEG): container finished" podID="36dc7329-32af-457b-aa1e-e0a38d67292a" containerID="28bcb00d2fcfa8221fdb5eb94fbc24788d01a2d821878cbf52d69276723e1da2" exitCode=0
Sep 11 22:00:55.713899 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:55.713541    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f4db57c75-fnkjr"
Sep 11 22:00:55.713899 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:55.713582    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f4db57c75-fnkjr" event={"ID":"36dc7329-32af-457b-aa1e-e0a38d67292a","Type":"ContainerDied","Data":"28bcb00d2fcfa8221fdb5eb94fbc24788d01a2d821878cbf52d69276723e1da2"}
Sep 11 22:00:55.713899 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:55.713644    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f4db57c75-fnkjr" event={"ID":"36dc7329-32af-457b-aa1e-e0a38d67292a","Type":"ContainerDied","Data":"049c9b8f09eb88f088982351c65cb415d4b584bdc3cd156c77e21acc3552b47e"}
Sep 11 22:00:55.713899 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:55.713673    2587 scope.go:117] "RemoveContainer" containerID="28bcb00d2fcfa8221fdb5eb94fbc24788d01a2d821878cbf52d69276723e1da2"
Sep 11 22:00:55.722140 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:55.722121    2587 scope.go:117] "RemoveContainer" containerID="28bcb00d2fcfa8221fdb5eb94fbc24788d01a2d821878cbf52d69276723e1da2"
Sep 11 22:00:55.722427 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:00:55.722410    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28bcb00d2fcfa8221fdb5eb94fbc24788d01a2d821878cbf52d69276723e1da2\": container with ID starting with 28bcb00d2fcfa8221fdb5eb94fbc24788d01a2d821878cbf52d69276723e1da2 not found: ID does not exist" containerID="28bcb00d2fcfa8221fdb5eb94fbc24788d01a2d821878cbf52d69276723e1da2"
Sep 11 22:00:55.722484 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:55.722456    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28bcb00d2fcfa8221fdb5eb94fbc24788d01a2d821878cbf52d69276723e1da2"} err="failed to get container status \"28bcb00d2fcfa8221fdb5eb94fbc24788d01a2d821878cbf52d69276723e1da2\": rpc error: code = NotFound desc = could not find container \"28bcb00d2fcfa8221fdb5eb94fbc24788d01a2d821878cbf52d69276723e1da2\": container with ID starting with 28bcb00d2fcfa8221fdb5eb94fbc24788d01a2d821878cbf52d69276723e1da2 not found: ID does not exist"
Sep 11 22:00:55.732134 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:55.732111    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f4db57c75-fnkjr"]
Sep 11 22:00:55.735938 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:55.735917    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5f4db57c75-fnkjr"]
Sep 11 22:00:57.335206 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:00:57.335176    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36dc7329-32af-457b-aa1e-e0a38d67292a" path="/var/lib/kubelet/pods/36dc7329-32af-457b-aa1e-e0a38d67292a/volumes"
Sep 11 22:01:16.250425 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.250387    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-758d76856b-8dm4c"]
Sep 11 22:01:16.250840 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.250758    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36dc7329-32af-457b-aa1e-e0a38d67292a" containerName="manager"
Sep 11 22:01:16.250840 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.250769    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="36dc7329-32af-457b-aa1e-e0a38d67292a" containerName="manager"
Sep 11 22:01:16.250840 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.250826    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="36dc7329-32af-457b-aa1e-e0a38d67292a" containerName="manager"
Sep 11 22:01:16.257119 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.257097    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c"
Sep 11 22:01:16.260513 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.260488    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 22:01:16.260638 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.260565    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 22:01:16.262562 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.262532    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mc9f9\""
Sep 11 22:01:16.262777 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.262758    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 22:01:16.262939 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.262919    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 22:01:16.273616 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.273589    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-758d76856b-8dm4c"]
Sep 11 22:01:16.288104 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.288079    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxpmj\" (UniqueName: \"kubernetes.io/projected/662be1ac-246e-43b9-a0b6-314e90293e1d-kube-api-access-cxpmj\") pod \"maas-api-758d76856b-8dm4c\" (UID: \"662be1ac-246e-43b9-a0b6-314e90293e1d\") " pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c"
Sep 11 22:01:16.288238 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.288131    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/662be1ac-246e-43b9-a0b6-314e90293e1d-maas-api-tls\") pod \"maas-api-758d76856b-8dm4c\" (UID: \"662be1ac-246e-43b9-a0b6-314e90293e1d\") " pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c"
Sep 11 22:01:16.288238 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.288183    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/662be1ac-246e-43b9-a0b6-314e90293e1d-openshift-service-ca\") pod \"maas-api-758d76856b-8dm4c\" (UID: \"662be1ac-246e-43b9-a0b6-314e90293e1d\") " pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c"
Sep 11 22:01:16.288238 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.288211    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/662be1ac-246e-43b9-a0b6-314e90293e1d-metrics-certs\") pod \"maas-api-758d76856b-8dm4c\" (UID: \"662be1ac-246e-43b9-a0b6-314e90293e1d\") " pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c"
Sep 11 22:01:16.389224 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.389188    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/662be1ac-246e-43b9-a0b6-314e90293e1d-openshift-service-ca\") pod \"maas-api-758d76856b-8dm4c\" (UID: \"662be1ac-246e-43b9-a0b6-314e90293e1d\") " pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c"
Sep 11 22:01:16.389430 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.389239    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/662be1ac-246e-43b9-a0b6-314e90293e1d-metrics-certs\") pod \"maas-api-758d76856b-8dm4c\" (UID: \"662be1ac-246e-43b9-a0b6-314e90293e1d\") " pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c"
Sep 11 22:01:16.389430 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.389305    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxpmj\" (UniqueName: \"kubernetes.io/projected/662be1ac-246e-43b9-a0b6-314e90293e1d-kube-api-access-cxpmj\") pod \"maas-api-758d76856b-8dm4c\" (UID: \"662be1ac-246e-43b9-a0b6-314e90293e1d\") " pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c"
Sep 11 22:01:16.389430 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.389419    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/662be1ac-246e-43b9-a0b6-314e90293e1d-maas-api-tls\") pod \"maas-api-758d76856b-8dm4c\" (UID: \"662be1ac-246e-43b9-a0b6-314e90293e1d\") " pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c"
Sep 11 22:01:16.389609 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:01:16.389421    2587 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 11 22:01:16.389609 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:01:16.389530    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/662be1ac-246e-43b9-a0b6-314e90293e1d-metrics-certs podName:662be1ac-246e-43b9-a0b6-314e90293e1d nodeName:}" failed. No retries permitted until 2026-09-11 22:01:16.889509031 +0000 UTC m=+530.064585380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/662be1ac-246e-43b9-a0b6-314e90293e1d-metrics-certs") pod "maas-api-758d76856b-8dm4c" (UID: "662be1ac-246e-43b9-a0b6-314e90293e1d") : secret "maas-api-metrics-tls" not found
Sep 11 22:01:16.390104 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.390079    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/662be1ac-246e-43b9-a0b6-314e90293e1d-openshift-service-ca\") pod \"maas-api-758d76856b-8dm4c\" (UID: \"662be1ac-246e-43b9-a0b6-314e90293e1d\") " pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c"
Sep 11 22:01:16.392091 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.392069    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/662be1ac-246e-43b9-a0b6-314e90293e1d-maas-api-tls\") pod \"maas-api-758d76856b-8dm4c\" (UID: \"662be1ac-246e-43b9-a0b6-314e90293e1d\") " pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c"
Sep 11 22:01:16.399490 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.399465    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxpmj\" (UniqueName: \"kubernetes.io/projected/662be1ac-246e-43b9-a0b6-314e90293e1d-kube-api-access-cxpmj\") pod \"maas-api-758d76856b-8dm4c\" (UID: \"662be1ac-246e-43b9-a0b6-314e90293e1d\") " pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c"
Sep 11 22:01:16.748653 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.748616    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-bsmwj"]
Sep 11 22:01:16.752311 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.752287    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-bsmwj"
Sep 11 22:01:16.755206 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.755180    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 11 22:01:16.755349 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.755181    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-w5hcf\""
Sep 11 22:01:16.763814 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.763790    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-bsmwj"]
Sep 11 22:01:16.792582 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.792550    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68c8f814-bf9c-4c1e-a642-741074aef1fd-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-bsmwj\" (UID: \"68c8f814-bf9c-4c1e-a642-741074aef1fd\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bsmwj"
Sep 11 22:01:16.792738 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.792589    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-254jb\" (UniqueName: \"kubernetes.io/projected/68c8f814-bf9c-4c1e-a642-741074aef1fd-kube-api-access-254jb\") pod \"payload-processing-5b54b7fbb7-bsmwj\" (UID: \"68c8f814-bf9c-4c1e-a642-741074aef1fd\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bsmwj"
Sep 11 22:01:16.892772 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.892729    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:01:16.893117 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.893087    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68c8f814-bf9c-4c1e-a642-741074aef1fd-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-bsmwj\" (UID: \"68c8f814-bf9c-4c1e-a642-741074aef1fd\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bsmwj"
Sep 11 22:01:16.893232 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.893146    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-254jb\" (UniqueName: \"kubernetes.io/projected/68c8f814-bf9c-4c1e-a642-741074aef1fd-kube-api-access-254jb\") pod \"payload-processing-5b54b7fbb7-bsmwj\" (UID: \"68c8f814-bf9c-4c1e-a642-741074aef1fd\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bsmwj"
Sep 11 22:01:16.893293 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.893244    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/662be1ac-246e-43b9-a0b6-314e90293e1d-metrics-certs\") pod \"maas-api-758d76856b-8dm4c\" (UID: \"662be1ac-246e-43b9-a0b6-314e90293e1d\") " pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c"
Sep 11 22:01:16.894023 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.893997    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68c8f814-bf9c-4c1e-a642-741074aef1fd-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-bsmwj\" (UID: \"68c8f814-bf9c-4c1e-a642-741074aef1fd\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bsmwj"
Sep 11 22:01:16.895802 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.895772    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/662be1ac-246e-43b9-a0b6-314e90293e1d-metrics-certs\") pod \"maas-api-758d76856b-8dm4c\" (UID: \"662be1ac-246e-43b9-a0b6-314e90293e1d\") " pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c"
Sep 11 22:01:16.907647 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:16.907618    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-254jb\" (UniqueName: \"kubernetes.io/projected/68c8f814-bf9c-4c1e-a642-741074aef1fd-kube-api-access-254jb\") pod \"payload-processing-5b54b7fbb7-bsmwj\" (UID: \"68c8f814-bf9c-4c1e-a642-741074aef1fd\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bsmwj"
Sep 11 22:01:17.064157 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:17.064071    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-bsmwj"
Sep 11 22:01:17.173289 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:17.173256    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c"
Sep 11 22:01:17.352611 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:17.352575    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-bsmwj"]
Sep 11 22:01:17.354910 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:01:17.354885    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c8f814_bf9c_4c1e_a642_741074aef1fd.slice/crio-642da92a30903bbfcf713e08c23aaf82354344ac662a0711781b827f3fdfba9d WatchSource:0}: Error finding container 642da92a30903bbfcf713e08c23aaf82354344ac662a0711781b827f3fdfba9d: Status 404 returned error can't find the container with id 642da92a30903bbfcf713e08c23aaf82354344ac662a0711781b827f3fdfba9d
Sep 11 22:01:17.465258 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:17.465231    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-758d76856b-8dm4c"]
Sep 11 22:01:17.466893 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:01:17.466856    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod662be1ac_246e_43b9_a0b6_314e90293e1d.slice/crio-376d2a9408341ea4e9989399d88b61c95a12c0a8608b6c19b7dd12963c4e19ae WatchSource:0}: Error finding container 376d2a9408341ea4e9989399d88b61c95a12c0a8608b6c19b7dd12963c4e19ae: Status 404 returned error can't find the container with id 376d2a9408341ea4e9989399d88b61c95a12c0a8608b6c19b7dd12963c4e19ae
Sep 11 22:01:17.816918 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:17.816882    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c" event={"ID":"662be1ac-246e-43b9-a0b6-314e90293e1d","Type":"ContainerStarted","Data":"376d2a9408341ea4e9989399d88b61c95a12c0a8608b6c19b7dd12963c4e19ae"}
Sep 11 22:01:17.818128 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:17.818104    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-bsmwj" event={"ID":"68c8f814-bf9c-4c1e-a642-741074aef1fd","Type":"ContainerStarted","Data":"642da92a30903bbfcf713e08c23aaf82354344ac662a0711781b827f3fdfba9d"}
Sep 11 22:01:19.849630 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:19.849516    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-bsmwj" event={"ID":"68c8f814-bf9c-4c1e-a642-741074aef1fd","Type":"ContainerStarted","Data":"b93df36d4d85d3e6682574f7f0361943145af7bce5c2234bbc5fca8d3f76e95b"}
Sep 11 22:01:19.849630 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:19.849596    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-bsmwj"
Sep 11 22:01:19.851346 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:19.851288    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c" event={"ID":"662be1ac-246e-43b9-a0b6-314e90293e1d","Type":"ContainerStarted","Data":"52b30a5db1a169418378dbf4915c0b36e483288c16e0d2a72c795cc0faa0f148"}
Sep 11 22:01:19.851475 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:19.851359    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c"
Sep 11 22:01:19.867235 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:19.867186    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-bsmwj" podStartSLOduration=1.763694748 podStartE2EDuration="3.867170157s" totalImagesPullingTime="2.103475409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:01:19.866101193 +0000 UTC m=+533.041177577" watchObservedRunningTime="2026-09-11 22:01:19.867170157 +0000 UTC m=+533.042246527"
Sep 11 22:01:19.882989 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:19.882937    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c" podStartSLOduration=1.887728627 podStartE2EDuration="3.882920051s" totalImagesPullingTime="1.995191424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:01:19.882034418 +0000 UTC m=+533.057110814" watchObservedRunningTime="2026-09-11 22:01:19.882920051 +0000 UTC m=+533.057996421"
Sep 11 22:01:25.861684 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:25.861586    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c"
Sep 11 22:01:25.883044 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:25.883013    2587 generic.go:356] "Generic (PLEG): container finished" podID="60203398-7954-425d-8699-a6c2bfd7bddb" containerID="c1e0a32a7d76604232000ca342abc36e7636bcb7d3f9c3f3f52f2accdcb182d1" exitCode=0
Sep 11 22:01:25.883192 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:25.883090    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7" event={"ID":"60203398-7954-425d-8699-a6c2bfd7bddb","Type":"ContainerDied","Data":"c1e0a32a7d76604232000ca342abc36e7636bcb7d3f9c3f3f52f2accdcb182d1"}
Sep 11 22:01:25.883502 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:25.883488    2587 scope.go:117] "RemoveContainer" containerID="c1e0a32a7d76604232000ca342abc36e7636bcb7d3f9c3f3f52f2accdcb182d1"
Sep 11 22:01:25.884626 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:25.884594    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:01:25.884751 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:25.884667    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:01:25.884751 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:25.884706    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:01:26.892009 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:26.891974    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7" event={"ID":"60203398-7954-425d-8699-a6c2bfd7bddb","Type":"ContainerStarted","Data":"d23474c9e6ffd4def4c4714348504447fe857212f19ffecb56fc938af3dfb501"}
Sep 11 22:01:27.736677 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:27.736636    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 22:01:27.738246 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:27.738213    2587 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-jtsp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 11 22:01:27.738554 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:27.738497    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7" podUID="60203398-7954-425d-8699-a6c2bfd7bddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 11 22:01:28.736852 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:28.736808    2587 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-jtsp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 11 22:01:28.737376 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:28.736895    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7" podUID="60203398-7954-425d-8699-a6c2bfd7bddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 11 22:01:29.737004 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:29.736961    2587 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-jtsp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 11 22:01:29.737664 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:29.737629    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7" podUID="60203398-7954-425d-8699-a6c2bfd7bddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 11 22:01:30.427122 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:30.427076    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 22:01:30.597922 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:30.597889    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:01:30.737599 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:30.737500    2587 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-jtsp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 11 22:01:30.738585 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:30.738148    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7" podUID="60203398-7954-425d-8699-a6c2bfd7bddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 11 22:01:30.859026 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:30.858998    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-bsmwj"
Sep 11 22:01:31.736847 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:31.736804    2587 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-jtsp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 11 22:01:31.737130 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:31.737098    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7" podUID="60203398-7954-425d-8699-a6c2bfd7bddb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 11 22:01:32.740844 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:32.740811    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 22:01:32.741742 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:32.741722    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-jtsp7"
Sep 11 22:01:46.586299 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:46.586252    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:01:53.576065 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:53.576028    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:01:59.871956 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:01:59.871916    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:02:12.584508 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.584471    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"]
Sep 11 22:02:12.588452 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.588427    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"
Sep 11 22:02:12.592727 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.592701    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 11 22:02:12.592879 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.592753    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 22:02:12.592879 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.592766    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 22:02:12.592879 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.592819    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-l4wd6\""
Sep 11 22:02:12.598902 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.598874    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"]
Sep 11 22:02:12.712523 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.712473    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ef083be-b3c6-41b8-b844-213b2fd574db-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg\" (UID: \"4ef083be-b3c6-41b8-b844-213b2fd574db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"
Sep 11 22:02:12.712722 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.712537    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ef083be-b3c6-41b8-b844-213b2fd574db-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg\" (UID: \"4ef083be-b3c6-41b8-b844-213b2fd574db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"
Sep 11 22:02:12.712722 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.712622    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9c7h\" (UniqueName: \"kubernetes.io/projected/4ef083be-b3c6-41b8-b844-213b2fd574db-kube-api-access-n9c7h\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg\" (UID: \"4ef083be-b3c6-41b8-b844-213b2fd574db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"
Sep 11 22:02:12.712722 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.712672    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ef083be-b3c6-41b8-b844-213b2fd574db-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg\" (UID: \"4ef083be-b3c6-41b8-b844-213b2fd574db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"
Sep 11 22:02:12.712828 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.712734    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ef083be-b3c6-41b8-b844-213b2fd574db-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg\" (UID: \"4ef083be-b3c6-41b8-b844-213b2fd574db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"
Sep 11 22:02:12.712828 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.712765    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ef083be-b3c6-41b8-b844-213b2fd574db-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg\" (UID: \"4ef083be-b3c6-41b8-b844-213b2fd574db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"
Sep 11 22:02:12.813730 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.813685    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ef083be-b3c6-41b8-b844-213b2fd574db-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg\" (UID: \"4ef083be-b3c6-41b8-b844-213b2fd574db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"
Sep 11 22:02:12.813730 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.813739    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ef083be-b3c6-41b8-b844-213b2fd574db-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg\" (UID: \"4ef083be-b3c6-41b8-b844-213b2fd574db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"
Sep 11 22:02:12.813987 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.813770    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9c7h\" (UniqueName: \"kubernetes.io/projected/4ef083be-b3c6-41b8-b844-213b2fd574db-kube-api-access-n9c7h\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg\" (UID: \"4ef083be-b3c6-41b8-b844-213b2fd574db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"
Sep 11 22:02:12.813987 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.813804    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ef083be-b3c6-41b8-b844-213b2fd574db-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg\" (UID: \"4ef083be-b3c6-41b8-b844-213b2fd574db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"
Sep 11 22:02:12.813987 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.813828    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ef083be-b3c6-41b8-b844-213b2fd574db-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg\" (UID: \"4ef083be-b3c6-41b8-b844-213b2fd574db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"
Sep 11 22:02:12.813987 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.813853    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ef083be-b3c6-41b8-b844-213b2fd574db-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg\" (UID: \"4ef083be-b3c6-41b8-b844-213b2fd574db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"
Sep 11 22:02:12.814284 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.814259    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ef083be-b3c6-41b8-b844-213b2fd574db-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg\" (UID: \"4ef083be-b3c6-41b8-b844-213b2fd574db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"
Sep 11 22:02:12.814284 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.814274    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ef083be-b3c6-41b8-b844-213b2fd574db-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg\" (UID: \"4ef083be-b3c6-41b8-b844-213b2fd574db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"
Sep 11 22:02:12.814430 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.814291    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ef083be-b3c6-41b8-b844-213b2fd574db-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg\" (UID: \"4ef083be-b3c6-41b8-b844-213b2fd574db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"
Sep 11 22:02:12.816099 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.816081    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ef083be-b3c6-41b8-b844-213b2fd574db-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg\" (UID: \"4ef083be-b3c6-41b8-b844-213b2fd574db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"
Sep 11 22:02:12.816409 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.816388    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ef083be-b3c6-41b8-b844-213b2fd574db-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg\" (UID: \"4ef083be-b3c6-41b8-b844-213b2fd574db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"
Sep 11 22:02:12.822728 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.822698    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9c7h\" (UniqueName: \"kubernetes.io/projected/4ef083be-b3c6-41b8-b844-213b2fd574db-kube-api-access-n9c7h\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg\" (UID: \"4ef083be-b3c6-41b8-b844-213b2fd574db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"
Sep 11 22:02:12.901891 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:12.901800    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"
Sep 11 22:02:13.175774 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:13.175749    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"]
Sep 11 22:02:13.178305 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:02:13.178275    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef083be_b3c6_41b8_b844_213b2fd574db.slice/crio-a5c110e0cfeb764506947275456af88c7178e8233f04c9e416710c2a6c79ac93 WatchSource:0}: Error finding container a5c110e0cfeb764506947275456af88c7178e8233f04c9e416710c2a6c79ac93: Status 404 returned error can't find the container with id a5c110e0cfeb764506947275456af88c7178e8233f04c9e416710c2a6c79ac93
Sep 11 22:02:14.124574 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:14.124537    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg" event={"ID":"4ef083be-b3c6-41b8-b844-213b2fd574db","Type":"ContainerStarted","Data":"a5c110e0cfeb764506947275456af88c7178e8233f04c9e416710c2a6c79ac93"}
Sep 11 22:02:14.170816 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:14.170777    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:02:15.137533 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:15.137485    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg" event={"ID":"4ef083be-b3c6-41b8-b844-213b2fd574db","Type":"ContainerStarted","Data":"d16372eee142e64c3041d4e5d4b40175e4843b5746b5484836c434d620b8bd85"}
Sep 11 22:02:15.156889 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:15.156816    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg" podStartSLOduration=2.179661042 podStartE2EDuration="3.156789415s" totalImagesPullingTime="977.128373ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:02:15.155217873 +0000 UTC m=+588.330294279" watchObservedRunningTime="2026-09-11 22:02:15.156789415 +0000 UTC m=+588.331865785"
Sep 11 22:02:21.167596 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:21.167567    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:02:22.902882 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:22.902847    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"
Sep 11 22:02:22.903315 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:22.902909    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"
Sep 11 22:02:22.915370 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:22.915323    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"
Sep 11 22:02:23.189485 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:23.189458    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg"
Sep 11 22:02:27.311664 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:27.311636    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw6qf_7ab8a820-c9ec-415c-bfaa-3c9744ac040a/ovn-acl-logging/0.log"
Sep 11 22:02:27.312090 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:27.311636    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw6qf_7ab8a820-c9ec-415c-bfaa-3c9744ac040a/ovn-acl-logging/0.log"
Sep 11 22:02:49.576476 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:02:49.576388    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:03:37.605684 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:37.605651    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-797db855dd-vt4lr"]
Sep 11 22:03:37.610598 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:37.610567    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-797db855dd-vt4lr"
Sep 11 22:03:37.616987 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:37.616957    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-797db855dd-vt4lr"]
Sep 11 22:03:37.695780 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:37.695752    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwsht\" (UniqueName: \"kubernetes.io/projected/30640515-d446-40a7-b8bb-f8aa52b448ba-kube-api-access-xwsht\") pod \"authorino-797db855dd-vt4lr\" (UID: \"30640515-d446-40a7-b8bb-f8aa52b448ba\") " pod="kuadrant-system/authorino-797db855dd-vt4lr"
Sep 11 22:03:37.695952 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:37.695795    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/30640515-d446-40a7-b8bb-f8aa52b448ba-tls-cert\") pod \"authorino-797db855dd-vt4lr\" (UID: \"30640515-d446-40a7-b8bb-f8aa52b448ba\") " pod="kuadrant-system/authorino-797db855dd-vt4lr"
Sep 11 22:03:37.796926 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:37.796891    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwsht\" (UniqueName: \"kubernetes.io/projected/30640515-d446-40a7-b8bb-f8aa52b448ba-kube-api-access-xwsht\") pod \"authorino-797db855dd-vt4lr\" (UID: \"30640515-d446-40a7-b8bb-f8aa52b448ba\") " pod="kuadrant-system/authorino-797db855dd-vt4lr"
Sep 11 22:03:37.797095 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:37.796942    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/30640515-d446-40a7-b8bb-f8aa52b448ba-tls-cert\") pod \"authorino-797db855dd-vt4lr\" (UID: \"30640515-d446-40a7-b8bb-f8aa52b448ba\") " pod="kuadrant-system/authorino-797db855dd-vt4lr"
Sep 11 22:03:37.799391 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:37.799369    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/30640515-d446-40a7-b8bb-f8aa52b448ba-tls-cert\") pod \"authorino-797db855dd-vt4lr\" (UID: \"30640515-d446-40a7-b8bb-f8aa52b448ba\") " pod="kuadrant-system/authorino-797db855dd-vt4lr"
Sep 11 22:03:37.804956 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:37.804932    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwsht\" (UniqueName: \"kubernetes.io/projected/30640515-d446-40a7-b8bb-f8aa52b448ba-kube-api-access-xwsht\") pod \"authorino-797db855dd-vt4lr\" (UID: \"30640515-d446-40a7-b8bb-f8aa52b448ba\") " pod="kuadrant-system/authorino-797db855dd-vt4lr"
Sep 11 22:03:37.923264 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:37.923227    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-797db855dd-vt4lr"
Sep 11 22:03:38.190205 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:38.190026    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-797db855dd-vt4lr"]
Sep 11 22:03:38.193013 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:03:38.192987    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30640515_d446_40a7_b8bb_f8aa52b448ba.slice/crio-6cc259e6a14a765836e137b51370a1a72582fb60c1b7d6fccc07192ab75dc8cc WatchSource:0}: Error finding container 6cc259e6a14a765836e137b51370a1a72582fb60c1b7d6fccc07192ab75dc8cc: Status 404 returned error can't find the container with id 6cc259e6a14a765836e137b51370a1a72582fb60c1b7d6fccc07192ab75dc8cc
Sep 11 22:03:38.540304 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:38.540212    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-797db855dd-vt4lr" event={"ID":"30640515-d446-40a7-b8bb-f8aa52b448ba","Type":"ContainerStarted","Data":"6cc259e6a14a765836e137b51370a1a72582fb60c1b7d6fccc07192ab75dc8cc"}
Sep 11 22:03:39.549584 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:39.549517    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-797db855dd-vt4lr" event={"ID":"30640515-d446-40a7-b8bb-f8aa52b448ba","Type":"ContainerStarted","Data":"bf92ab2b304ff9f2550c49e144838f8a6f4546798cea455fa1c52dfdab10abf4"}
Sep 11 22:03:39.567222 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:39.567169    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-797db855dd-vt4lr" podStartSLOduration=1.903839347 podStartE2EDuration="2.567150859s" totalImagesPullingTime="663.311512ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:03:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:03:39.566002245 +0000 UTC m=+672.741078617" watchObservedRunningTime="2026-09-11 22:03:39.567150859 +0000 UTC m=+672.742227228"
Sep 11 22:03:39.597569 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:39.597528    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f55d54cdf-26nvb"]
Sep 11 22:03:39.597755 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:39.597733    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-f55d54cdf-26nvb" podUID="3264d17b-c9fd-4959-a492-a368e7c17249" containerName="authorino" containerID="cri-o://a8d3cade4c6e9a7a748c4c2961e153dd0510d8452dd0472da22d3c6035fef9a2" gracePeriod=30
Sep 11 22:03:39.862920 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:39.862894    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f55d54cdf-26nvb"
Sep 11 22:03:39.917824 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:39.917783    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-774bp\" (UniqueName: \"kubernetes.io/projected/3264d17b-c9fd-4959-a492-a368e7c17249-kube-api-access-774bp\") pod \"3264d17b-c9fd-4959-a492-a368e7c17249\" (UID: \"3264d17b-c9fd-4959-a492-a368e7c17249\") "
Sep 11 22:03:39.917988 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:39.917973    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3264d17b-c9fd-4959-a492-a368e7c17249-tls-cert\") pod \"3264d17b-c9fd-4959-a492-a368e7c17249\" (UID: \"3264d17b-c9fd-4959-a492-a368e7c17249\") "
Sep 11 22:03:39.919833 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:39.919803    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3264d17b-c9fd-4959-a492-a368e7c17249-kube-api-access-774bp" (OuterVolumeSpecName: "kube-api-access-774bp") pod "3264d17b-c9fd-4959-a492-a368e7c17249" (UID: "3264d17b-c9fd-4959-a492-a368e7c17249"). InnerVolumeSpecName "kube-api-access-774bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:03:39.928510 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:39.928482    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3264d17b-c9fd-4959-a492-a368e7c17249-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "3264d17b-c9fd-4959-a492-a368e7c17249" (UID: "3264d17b-c9fd-4959-a492-a368e7c17249"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:03:40.018907 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:40.018860    2587 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3264d17b-c9fd-4959-a492-a368e7c17249-tls-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:03:40.018907 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:40.018904    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-774bp\" (UniqueName: \"kubernetes.io/projected/3264d17b-c9fd-4959-a492-a368e7c17249-kube-api-access-774bp\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:03:40.556026 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:40.555989    2587 generic.go:356] "Generic (PLEG): container finished" podID="3264d17b-c9fd-4959-a492-a368e7c17249" containerID="a8d3cade4c6e9a7a748c4c2961e153dd0510d8452dd0472da22d3c6035fef9a2" exitCode=0
Sep 11 22:03:40.556473 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:40.556034    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f55d54cdf-26nvb"
Sep 11 22:03:40.556473 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:40.556074    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f55d54cdf-26nvb" event={"ID":"3264d17b-c9fd-4959-a492-a368e7c17249","Type":"ContainerDied","Data":"a8d3cade4c6e9a7a748c4c2961e153dd0510d8452dd0472da22d3c6035fef9a2"}
Sep 11 22:03:40.556473 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:40.556121    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f55d54cdf-26nvb" event={"ID":"3264d17b-c9fd-4959-a492-a368e7c17249","Type":"ContainerDied","Data":"d916db52be59169634118d31f61cebc9ef690529970bc5dc49b442959d979d0b"}
Sep 11 22:03:40.556473 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:40.556142    2587 scope.go:117] "RemoveContainer" containerID="a8d3cade4c6e9a7a748c4c2961e153dd0510d8452dd0472da22d3c6035fef9a2"
Sep 11 22:03:40.565545 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:40.565520    2587 scope.go:117] "RemoveContainer" containerID="a8d3cade4c6e9a7a748c4c2961e153dd0510d8452dd0472da22d3c6035fef9a2"
Sep 11 22:03:40.565865 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:03:40.565848    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8d3cade4c6e9a7a748c4c2961e153dd0510d8452dd0472da22d3c6035fef9a2\": container with ID starting with a8d3cade4c6e9a7a748c4c2961e153dd0510d8452dd0472da22d3c6035fef9a2 not found: ID does not exist" containerID="a8d3cade4c6e9a7a748c4c2961e153dd0510d8452dd0472da22d3c6035fef9a2"
Sep 11 22:03:40.565911 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:40.565875    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8d3cade4c6e9a7a748c4c2961e153dd0510d8452dd0472da22d3c6035fef9a2"} err="failed to get container status \"a8d3cade4c6e9a7a748c4c2961e153dd0510d8452dd0472da22d3c6035fef9a2\": rpc error: code = NotFound desc = could not find container \"a8d3cade4c6e9a7a748c4c2961e153dd0510d8452dd0472da22d3c6035fef9a2\": container with ID starting with a8d3cade4c6e9a7a748c4c2961e153dd0510d8452dd0472da22d3c6035fef9a2 not found: ID does not exist"
Sep 11 22:03:40.579529 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:40.579495    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f55d54cdf-26nvb"]
Sep 11 22:03:40.581996 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:40.581973    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-f55d54cdf-26nvb"]
Sep 11 22:03:41.336291 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:41.336252    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3264d17b-c9fd-4959-a492-a368e7c17249" path="/var/lib/kubelet/pods/3264d17b-c9fd-4959-a492-a368e7c17249/volumes"
Sep 11 22:03:56.999525 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:56.999476    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"]
Sep 11 22:03:56.999997 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:56.999950    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3264d17b-c9fd-4959-a492-a368e7c17249" containerName="authorino"
Sep 11 22:03:56.999997 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:56.999966    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="3264d17b-c9fd-4959-a492-a368e7c17249" containerName="authorino"
Sep 11 22:03:57.000111 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.000070    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="3264d17b-c9fd-4959-a492-a368e7c17249" containerName="authorino"
Sep 11 22:03:57.003323 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.003293    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.006410 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.006376    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-27dad9-openshift-default-dockercfg-w7dpl\""
Sep 11 22:03:57.017170 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.017144    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"]
Sep 11 22:03:57.078910 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.078871    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2w4c\" (UniqueName: \"kubernetes.io/projected/f2f9ee75-e2ab-45e6-9f74-9088f1508251-kube-api-access-q2w4c\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.079081 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.078926    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-workload-socket\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.079081 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.078953    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-workload-certs\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.079081 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.078980    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istio-data\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.079081 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.078998    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istio-token\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.079081 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.079015    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istiod-ca-cert\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.079081 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.079070    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-credential-socket\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.080040 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.079127    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istio-envoy\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.080040 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.079161    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istio-podinfo\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.146244 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.146210    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"]
Sep 11 22:03:57.154717 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.154685    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.158003 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.157980    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-95d6c2-openshift-default-dockercfg-d6kwf\""
Sep 11 22:03:57.167997 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.167971    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"]
Sep 11 22:03:57.180662 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.180362    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istio-envoy\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.181020 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.180410    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istio-podinfo\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.181290 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.181247    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2w4c\" (UniqueName: \"kubernetes.io/projected/f2f9ee75-e2ab-45e6-9f74-9088f1508251-kube-api-access-q2w4c\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.181503 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.181405    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-workload-socket\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.181503 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.181458    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-workload-certs\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.181738 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.181685    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istio-data\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.181897 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.181717    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istio-token\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.181897 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.181843    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istiod-ca-cert\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.182209 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.182110    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-credential-socket\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.183137 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.183100    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-credential-socket\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.184511 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.183555    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istio-data\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.184511 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.184398    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-workload-certs\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.184982 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.184947    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istio-envoy\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.185557 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.185539    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-workload-socket\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.186029 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.185983    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istiod-ca-cert\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.190998 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.190971    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istio-podinfo\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.192083 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.192033    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2w4c\" (UniqueName: \"kubernetes.io/projected/f2f9ee75-e2ab-45e6-9f74-9088f1508251-kube-api-access-q2w4c\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.193529 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.193506    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istio-token\") pod \"e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") " pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.282833 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.282730    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe4daef-73cf-4488-acec-a2b20a88db5e-istio-podinfo\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.282833 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.282785    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fe4daef-73cf-4488-acec-a2b20a88db5e-istio-token\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.283067 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.282863    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-workload-certs\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.283067 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.282909    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-istio-data\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.283067 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.282943    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm5hv\" (UniqueName: \"kubernetes.io/projected/9fe4daef-73cf-4488-acec-a2b20a88db5e-kube-api-access-sm5hv\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.283067 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.283045    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-credential-socket\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.283306 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.283071    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-istio-envoy\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.283306 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.283095    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe4daef-73cf-4488-acec-a2b20a88db5e-istiod-ca-cert\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.283306 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.283126    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-workload-socket\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.316477 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.316443    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:57.388184 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.383901    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fe4daef-73cf-4488-acec-a2b20a88db5e-istio-token\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.388184 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.383975    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-workload-certs\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.388184 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.384005    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-istio-data\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.388184 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.384035    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm5hv\" (UniqueName: \"kubernetes.io/projected/9fe4daef-73cf-4488-acec-a2b20a88db5e-kube-api-access-sm5hv\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.388184 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.384139    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-credential-socket\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.388184 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.384165    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-istio-envoy\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.388184 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.384192    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe4daef-73cf-4488-acec-a2b20a88db5e-istiod-ca-cert\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.388184 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.384226    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-workload-socket\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.388184 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.384286    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe4daef-73cf-4488-acec-a2b20a88db5e-istio-podinfo\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.388184 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.385283    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-workload-certs\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.388184 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.385582    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-istio-data\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.388184 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.385630    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-credential-socket\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.388184 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.386903    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe4daef-73cf-4488-acec-a2b20a88db5e-istiod-ca-cert\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.388184 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.387425    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-workload-socket\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.395616 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.395147    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-istio-envoy\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.398214 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.398166    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm5hv\" (UniqueName: \"kubernetes.io/projected/9fe4daef-73cf-4488-acec-a2b20a88db5e-kube-api-access-sm5hv\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.400293 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.400102    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe4daef-73cf-4488-acec-a2b20a88db5e-istio-podinfo\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.400293 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.400251    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fe4daef-73cf-4488-acec-a2b20a88db5e-istio-token\") pod \"e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") " pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.470212 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.470174    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:57.606745 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.606719    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"]
Sep 11 22:03:57.607986 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:03:57.607937    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f9ee75_e2ab_45e6_9f74_9088f1508251.slice/crio-c570cc0b3880e2bdce62e260dee3814b8568991c32ea8d073136d31e0021a7a7 WatchSource:0}: Error finding container c570cc0b3880e2bdce62e260dee3814b8568991c32ea8d073136d31e0021a7a7: Status 404 returned error can't find the container with id c570cc0b3880e2bdce62e260dee3814b8568991c32ea8d073136d31e0021a7a7
Sep 11 22:03:57.610291 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.610242    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:03:57.610408 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.610360    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:03:57.610469 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.610408    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:03:57.653985 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.653951    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq" event={"ID":"f2f9ee75-e2ab-45e6-9f74-9088f1508251","Type":"ContainerStarted","Data":"c570cc0b3880e2bdce62e260dee3814b8568991c32ea8d073136d31e0021a7a7"}
Sep 11 22:03:57.755156 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.755128    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"]
Sep 11 22:03:57.756501 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:03:57.756474    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe4daef_73cf_4488_acec_a2b20a88db5e.slice/crio-d363556873a292a1abf6b40f87c223ce9201e488f403e86f4c6f1f543af02a44 WatchSource:0}: Error finding container d363556873a292a1abf6b40f87c223ce9201e488f403e86f4c6f1f543af02a44: Status 404 returned error can't find the container with id d363556873a292a1abf6b40f87c223ce9201e488f403e86f4c6f1f543af02a44
Sep 11 22:03:57.759180 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.759143    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:03:57.759264 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.759220    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:03:57.759264 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:57.759249    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:03:58.661454 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:58.661407    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq" event={"ID":"f2f9ee75-e2ab-45e6-9f74-9088f1508251","Type":"ContainerStarted","Data":"d8c1f7b71fe6572df3397251cfd4a567d0ec3d9fabd779d5cae1031ff5b00699"}
Sep 11 22:03:58.663080 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:58.663051    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk" event={"ID":"9fe4daef-73cf-4488-acec-a2b20a88db5e","Type":"ContainerStarted","Data":"1d9b78d1272c6daf4a1dfb776ce66888821506805c8d7a20cf42c85144f0751b"}
Sep 11 22:03:58.663182 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:58.663101    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk" event={"ID":"9fe4daef-73cf-4488-acec-a2b20a88db5e","Type":"ContainerStarted","Data":"d363556873a292a1abf6b40f87c223ce9201e488f403e86f4c6f1f543af02a44"}
Sep 11 22:03:58.685303 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:58.685250    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq" podStartSLOduration=2.685232946 podStartE2EDuration="2.685232946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:03:58.684174507 +0000 UTC m=+691.859250908" watchObservedRunningTime="2026-09-11 22:03:58.685232946 +0000 UTC m=+691.860309316"
Sep 11 22:03:58.709147 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:58.709100    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk" podStartSLOduration=1.7090854260000001 podStartE2EDuration="1.709085426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:03:58.707762408 +0000 UTC m=+691.882838777" watchObservedRunningTime="2026-09-11 22:03:58.709085426 +0000 UTC m=+691.884161794"
Sep 11 22:03:59.317307 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:59.317273    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:59.322966 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:59.322938    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:59.470740 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:59.470710    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:59.475855 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:59.475831    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:59.668796 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:59.668758    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:03:59.669232 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:59.668815    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:59.669695 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:59.669668    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:03:59.669901 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:03:59.669879    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:04:00.391772 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:00.391727    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:04:01.577043 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:01.577010    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5"]
Sep 11 22:04:01.602800 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:01.599479    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5"
Sep 11 22:04:01.603568 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:01.603527    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5"]
Sep 11 22:04:01.604491 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:01.604461    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-7e2add-serving-cert\""
Sep 11 22:04:01.729279 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:01.729241    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42c7eecb-f974-4b7d-aea5-2190e35a35f4-maas-api-tls\") pod \"maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5\" (UID: \"42c7eecb-f974-4b7d-aea5-2190e35a35f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5"
Sep 11 22:04:01.729465 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:01.729317    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c7eecb-f974-4b7d-aea5-2190e35a35f4-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5\" (UID: \"42c7eecb-f974-4b7d-aea5-2190e35a35f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5"
Sep 11 22:04:01.729465 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:01.729360    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42c7eecb-f974-4b7d-aea5-2190e35a35f4-metrics-certs\") pod \"maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5\" (UID: \"42c7eecb-f974-4b7d-aea5-2190e35a35f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5"
Sep 11 22:04:01.729559 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:01.729472    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z59tj\" (UniqueName: \"kubernetes.io/projected/42c7eecb-f974-4b7d-aea5-2190e35a35f4-kube-api-access-z59tj\") pod \"maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5\" (UID: \"42c7eecb-f974-4b7d-aea5-2190e35a35f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5"
Sep 11 22:04:01.830629 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:01.830530    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c7eecb-f974-4b7d-aea5-2190e35a35f4-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5\" (UID: \"42c7eecb-f974-4b7d-aea5-2190e35a35f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5"
Sep 11 22:04:01.830629 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:01.830590    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42c7eecb-f974-4b7d-aea5-2190e35a35f4-metrics-certs\") pod \"maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5\" (UID: \"42c7eecb-f974-4b7d-aea5-2190e35a35f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5"
Sep 11 22:04:01.830865 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:01.830698    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z59tj\" (UniqueName: \"kubernetes.io/projected/42c7eecb-f974-4b7d-aea5-2190e35a35f4-kube-api-access-z59tj\") pod \"maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5\" (UID: \"42c7eecb-f974-4b7d-aea5-2190e35a35f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5"
Sep 11 22:04:01.830925 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:01.830857    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42c7eecb-f974-4b7d-aea5-2190e35a35f4-maas-api-tls\") pod \"maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5\" (UID: \"42c7eecb-f974-4b7d-aea5-2190e35a35f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5"
Sep 11 22:04:01.831788 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:04:01.830981    2587 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-serving-cert: secret "maas-api-e2e-worker-w4-7e2add-serving-cert" not found
Sep 11 22:04:01.831788 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:04:01.831045    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42c7eecb-f974-4b7d-aea5-2190e35a35f4-maas-api-tls podName:42c7eecb-f974-4b7d-aea5-2190e35a35f4 nodeName:}" failed. No retries permitted until 2026-09-11 22:04:02.331026015 +0000 UTC m=+695.506102382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/42c7eecb-f974-4b7d-aea5-2190e35a35f4-maas-api-tls") pod "maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5" (UID: "42c7eecb-f974-4b7d-aea5-2190e35a35f4") : secret "maas-api-e2e-worker-w4-7e2add-serving-cert" not found
Sep 11 22:04:01.831788 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:01.831292    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c7eecb-f974-4b7d-aea5-2190e35a35f4-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5\" (UID: \"42c7eecb-f974-4b7d-aea5-2190e35a35f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5"
Sep 11 22:04:01.833963 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:01.833941    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42c7eecb-f974-4b7d-aea5-2190e35a35f4-metrics-certs\") pod \"maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5\" (UID: \"42c7eecb-f974-4b7d-aea5-2190e35a35f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5"
Sep 11 22:04:01.840430 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:01.840396    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z59tj\" (UniqueName: \"kubernetes.io/projected/42c7eecb-f974-4b7d-aea5-2190e35a35f4-kube-api-access-z59tj\") pod \"maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5\" (UID: \"42c7eecb-f974-4b7d-aea5-2190e35a35f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5"
Sep 11 22:04:02.335462 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:02.335430    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42c7eecb-f974-4b7d-aea5-2190e35a35f4-maas-api-tls\") pod \"maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5\" (UID: \"42c7eecb-f974-4b7d-aea5-2190e35a35f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5"
Sep 11 22:04:02.337928 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:02.337906    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42c7eecb-f974-4b7d-aea5-2190e35a35f4-maas-api-tls\") pod \"maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5\" (UID: \"42c7eecb-f974-4b7d-aea5-2190e35a35f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5"
Sep 11 22:04:02.515992 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:02.515957    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5"
Sep 11 22:04:02.693654 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:02.693620    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6"]
Sep 11 22:04:02.699071 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:02.699046    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6"
Sep 11 22:04:02.702255 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:02.702234    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-95d6c2-serving-cert\""
Sep 11 22:04:02.709538 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:02.709473    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6"]
Sep 11 22:04:02.807125 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:02.807100    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5"]
Sep 11 22:04:02.810851 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:04:02.810814    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c7eecb_f974_4b7d_aea5_2190e35a35f4.slice/crio-8a6a04aa0c5b9f8f4518d477bb92cedb33908d7ce209f5f9e01179aca23b3eb1 WatchSource:0}: Error finding container 8a6a04aa0c5b9f8f4518d477bb92cedb33908d7ce209f5f9e01179aca23b3eb1: Status 404 returned error can't find the container with id 8a6a04aa0c5b9f8f4518d477bb92cedb33908d7ce209f5f9e01179aca23b3eb1
Sep 11 22:04:02.811661 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:02.811642    2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 22:04:02.841607 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:02.841577    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54e64fba-6ece-4dee-8e5f-76210f2c4ed8-metrics-certs\") pod \"maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6\" (UID: \"54e64fba-6ece-4dee-8e5f-76210f2c4ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6"
Sep 11 22:04:02.841750 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:02.841686    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e64fba-6ece-4dee-8e5f-76210f2c4ed8-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6\" (UID: \"54e64fba-6ece-4dee-8e5f-76210f2c4ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6"
Sep 11 22:04:02.841834 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:02.841761    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54e64fba-6ece-4dee-8e5f-76210f2c4ed8-maas-api-tls\") pod \"maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6\" (UID: \"54e64fba-6ece-4dee-8e5f-76210f2c4ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6"
Sep 11 22:04:02.841891 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:02.841832    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8jv2\" (UniqueName: \"kubernetes.io/projected/54e64fba-6ece-4dee-8e5f-76210f2c4ed8-kube-api-access-q8jv2\") pod \"maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6\" (UID: \"54e64fba-6ece-4dee-8e5f-76210f2c4ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6"
Sep 11 22:04:02.942812 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:02.942773    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8jv2\" (UniqueName: \"kubernetes.io/projected/54e64fba-6ece-4dee-8e5f-76210f2c4ed8-kube-api-access-q8jv2\") pod \"maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6\" (UID: \"54e64fba-6ece-4dee-8e5f-76210f2c4ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6"
Sep 11 22:04:02.942989 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:02.942835    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54e64fba-6ece-4dee-8e5f-76210f2c4ed8-metrics-certs\") pod \"maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6\" (UID: \"54e64fba-6ece-4dee-8e5f-76210f2c4ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6"
Sep 11 22:04:02.942989 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:02.942915    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e64fba-6ece-4dee-8e5f-76210f2c4ed8-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6\" (UID: \"54e64fba-6ece-4dee-8e5f-76210f2c4ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6"
Sep 11 22:04:02.942989 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:02.942958    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54e64fba-6ece-4dee-8e5f-76210f2c4ed8-maas-api-tls\") pod \"maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6\" (UID: \"54e64fba-6ece-4dee-8e5f-76210f2c4ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6"
Sep 11 22:04:02.943735 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:02.943675    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e64fba-6ece-4dee-8e5f-76210f2c4ed8-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6\" (UID: \"54e64fba-6ece-4dee-8e5f-76210f2c4ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6"
Sep 11 22:04:02.945642 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:02.945617    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54e64fba-6ece-4dee-8e5f-76210f2c4ed8-metrics-certs\") pod \"maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6\" (UID: \"54e64fba-6ece-4dee-8e5f-76210f2c4ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6"
Sep 11 22:04:02.945734 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:02.945654    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54e64fba-6ece-4dee-8e5f-76210f2c4ed8-maas-api-tls\") pod \"maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6\" (UID: \"54e64fba-6ece-4dee-8e5f-76210f2c4ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6"
Sep 11 22:04:02.952193 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:02.952168    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8jv2\" (UniqueName: \"kubernetes.io/projected/54e64fba-6ece-4dee-8e5f-76210f2c4ed8-kube-api-access-q8jv2\") pod \"maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6\" (UID: \"54e64fba-6ece-4dee-8e5f-76210f2c4ed8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6"
Sep 11 22:04:03.015212 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:03.015183    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6"
Sep 11 22:04:03.296752 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:03.296677    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6"]
Sep 11 22:04:03.298840 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:04:03.298814    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e64fba_6ece_4dee_8e5f_76210f2c4ed8.slice/crio-0f256e74392aac074aef81b1c1eb3998df2be2193fef27309c34c257abde5049 WatchSource:0}: Error finding container 0f256e74392aac074aef81b1c1eb3998df2be2193fef27309c34c257abde5049: Status 404 returned error can't find the container with id 0f256e74392aac074aef81b1c1eb3998df2be2193fef27309c34c257abde5049
Sep 11 22:04:03.682847 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:03.682575    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz"]
Sep 11 22:04:03.687841 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:03.687819    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz"
Sep 11 22:04:03.690514 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:03.690490    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-27dad9-serving-cert\""
Sep 11 22:04:03.696583 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:03.696546    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz"]
Sep 11 22:04:03.700310 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:03.700276    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5" event={"ID":"42c7eecb-f974-4b7d-aea5-2190e35a35f4","Type":"ContainerStarted","Data":"4d66f8e1aa01b8f884a9263a2be884815ca92e43bb2cff00442dec667ffb3570"}
Sep 11 22:04:03.700469 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:03.700354    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5" event={"ID":"42c7eecb-f974-4b7d-aea5-2190e35a35f4","Type":"ContainerStarted","Data":"8a6a04aa0c5b9f8f4518d477bb92cedb33908d7ce209f5f9e01179aca23b3eb1"}
Sep 11 22:04:03.700889 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:03.700871    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5"
Sep 11 22:04:03.701929 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:03.701903    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6" event={"ID":"54e64fba-6ece-4dee-8e5f-76210f2c4ed8","Type":"ContainerStarted","Data":"0f256e74392aac074aef81b1c1eb3998df2be2193fef27309c34c257abde5049"}
Sep 11 22:04:03.727799 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:03.727743    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5" podStartSLOduration=2.3027547569999998 podStartE2EDuration="2.727726386s" totalImagesPullingTime="424.971629ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:03.725672346 +0000 UTC m=+696.900748745" watchObservedRunningTime="2026-09-11 22:04:03.727726386 +0000 UTC m=+696.902802755"
Sep 11 22:04:03.852469 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:03.852422    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf-maas-api-tls\") pod \"maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz\" (UID: \"9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz"
Sep 11 22:04:03.852624 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:03.852509    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz\" (UID: \"9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz"
Sep 11 22:04:03.852624 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:03.852598    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf-metrics-certs\") pod \"maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz\" (UID: \"9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz"
Sep 11 22:04:03.852753 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:03.852647    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r78t5\" (UniqueName: \"kubernetes.io/projected/9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf-kube-api-access-r78t5\") pod \"maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz\" (UID: \"9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz"
Sep 11 22:04:03.954098 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:03.954006    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz\" (UID: \"9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz"
Sep 11 22:04:03.954098 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:03.954059    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf-metrics-certs\") pod \"maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz\" (UID: \"9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz"
Sep 11 22:04:03.954322 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:03.954200    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r78t5\" (UniqueName: \"kubernetes.io/projected/9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf-kube-api-access-r78t5\") pod \"maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz\" (UID: \"9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz"
Sep 11 22:04:03.954322 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:03.954314    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf-maas-api-tls\") pod \"maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz\" (UID: \"9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz"
Sep 11 22:04:03.954844 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:03.954819    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz\" (UID: \"9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz"
Sep 11 22:04:03.956915 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:03.956888    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf-maas-api-tls\") pod \"maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz\" (UID: \"9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz"
Sep 11 22:04:03.957080 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:03.957056    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf-metrics-certs\") pod \"maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz\" (UID: \"9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz"
Sep 11 22:04:03.963953 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:03.963923    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r78t5\" (UniqueName: \"kubernetes.io/projected/9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf-kube-api-access-r78t5\") pod \"maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz\" (UID: \"9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz"
Sep 11 22:04:04.082987 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:04.082950    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz"
Sep 11 22:04:04.361651 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:04.361624    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz"]
Sep 11 22:04:04.363625 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:04:04.363602    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d21f55a_2bfb_4ffa_baf0_d3eb359b67bf.slice/crio-898acfe967a4254063e864af858e531c4950fe2e35562fa7e59cc7fcb810a86e WatchSource:0}: Error finding container 898acfe967a4254063e864af858e531c4950fe2e35562fa7e59cc7fcb810a86e: Status 404 returned error can't find the container with id 898acfe967a4254063e864af858e531c4950fe2e35562fa7e59cc7fcb810a86e
Sep 11 22:04:04.709395 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:04.709357    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz" event={"ID":"9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf","Type":"ContainerStarted","Data":"898acfe967a4254063e864af858e531c4950fe2e35562fa7e59cc7fcb810a86e"}
Sep 11 22:04:04.711243 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:04.711213    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6" event={"ID":"54e64fba-6ece-4dee-8e5f-76210f2c4ed8","Type":"ContainerStarted","Data":"0c4be844dc02c1cd4e21acea6a8eb88da230f85ae69411e4db8ffb4104b93dd3"}
Sep 11 22:04:04.711502 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:04.711483    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6"
Sep 11 22:04:04.732218 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:04.732171    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6" podStartSLOduration=2.362700596 podStartE2EDuration="2.732154823s" totalImagesPullingTime="369.454227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:04.73087472 +0000 UTC m=+697.905951122" watchObservedRunningTime="2026-09-11 22:04:04.732154823 +0000 UTC m=+697.907231193"
Sep 11 22:04:05.721016 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:05.720075    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz" event={"ID":"9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf","Type":"ContainerStarted","Data":"18144a86bed2402eea8707ecc7f330f9b61adcacc35b67e29f35dea5ae42a8b4"}
Sep 11 22:04:05.721016 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:05.720404    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz"
Sep 11 22:04:05.746550 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:05.746493    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz" podStartSLOduration=2.369421766 podStartE2EDuration="2.746475726s" totalImagesPullingTime="377.05396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:05.744898223 +0000 UTC m=+698.919974603" watchObservedRunningTime="2026-09-11 22:04:05.746475726 +0000 UTC m=+698.921552123"
Sep 11 22:04:09.717208 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:09.717169    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5"
Sep 11 22:04:10.730368 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:10.730313    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6"
Sep 11 22:04:11.743668 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:11.743627    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz"
Sep 11 22:04:13.638793 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:13.638755    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp"]
Sep 11 22:04:13.654081 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:13.653901    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp"
Sep 11 22:04:13.657626 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:13.657602    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-7d5ec8-serving-cert\""
Sep 11 22:04:13.668347 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:13.662664    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp"]
Sep 11 22:04:13.680341 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:13.680256    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13929af-4c3f-42f2-9179-716de50abacb-openshift-service-ca\") pod \"maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp\" (UID: \"c13929af-4c3f-42f2-9179-716de50abacb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp"
Sep 11 22:04:13.680341 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:13.680315    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c13929af-4c3f-42f2-9179-716de50abacb-metrics-certs\") pod \"maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp\" (UID: \"c13929af-4c3f-42f2-9179-716de50abacb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp"
Sep 11 22:04:13.680543 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:13.680378    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q2gr\" (UniqueName: \"kubernetes.io/projected/c13929af-4c3f-42f2-9179-716de50abacb-kube-api-access-9q2gr\") pod \"maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp\" (UID: \"c13929af-4c3f-42f2-9179-716de50abacb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp"
Sep 11 22:04:13.680543 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:13.680439    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c13929af-4c3f-42f2-9179-716de50abacb-maas-api-tls\") pod \"maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp\" (UID: \"c13929af-4c3f-42f2-9179-716de50abacb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp"
Sep 11 22:04:13.781901 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:13.781171    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13929af-4c3f-42f2-9179-716de50abacb-openshift-service-ca\") pod \"maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp\" (UID: \"c13929af-4c3f-42f2-9179-716de50abacb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp"
Sep 11 22:04:13.781901 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:13.781240    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c13929af-4c3f-42f2-9179-716de50abacb-metrics-certs\") pod \"maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp\" (UID: \"c13929af-4c3f-42f2-9179-716de50abacb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp"
Sep 11 22:04:13.781901 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:13.781295    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q2gr\" (UniqueName: \"kubernetes.io/projected/c13929af-4c3f-42f2-9179-716de50abacb-kube-api-access-9q2gr\") pod \"maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp\" (UID: \"c13929af-4c3f-42f2-9179-716de50abacb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp"
Sep 11 22:04:13.781901 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:13.781377    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c13929af-4c3f-42f2-9179-716de50abacb-maas-api-tls\") pod \"maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp\" (UID: \"c13929af-4c3f-42f2-9179-716de50abacb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp"
Sep 11 22:04:13.781901 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:04:13.781527    2587 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-serving-cert: secret "maas-api-e2e-autores-7d5ec8-serving-cert" not found
Sep 11 22:04:13.781901 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:04:13.781600    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c13929af-4c3f-42f2-9179-716de50abacb-maas-api-tls podName:c13929af-4c3f-42f2-9179-716de50abacb nodeName:}" failed. No retries permitted until 2026-09-11 22:04:14.281578278 +0000 UTC m=+707.456654628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c13929af-4c3f-42f2-9179-716de50abacb-maas-api-tls") pod "maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp" (UID: "c13929af-4c3f-42f2-9179-716de50abacb") : secret "maas-api-e2e-autores-7d5ec8-serving-cert" not found
Sep 11 22:04:13.782404 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:13.782100    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13929af-4c3f-42f2-9179-716de50abacb-openshift-service-ca\") pod \"maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp\" (UID: \"c13929af-4c3f-42f2-9179-716de50abacb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp"
Sep 11 22:04:13.784061 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:13.784038    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c13929af-4c3f-42f2-9179-716de50abacb-metrics-certs\") pod \"maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp\" (UID: \"c13929af-4c3f-42f2-9179-716de50abacb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp"
Sep 11 22:04:13.794205 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:13.794179    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q2gr\" (UniqueName: \"kubernetes.io/projected/c13929af-4c3f-42f2-9179-716de50abacb-kube-api-access-9q2gr\") pod \"maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp\" (UID: \"c13929af-4c3f-42f2-9179-716de50abacb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp"
Sep 11 22:04:14.286386 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:14.286349    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c13929af-4c3f-42f2-9179-716de50abacb-maas-api-tls\") pod \"maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp\" (UID: \"c13929af-4c3f-42f2-9179-716de50abacb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp"
Sep 11 22:04:14.290909 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:14.289300    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c13929af-4c3f-42f2-9179-716de50abacb-maas-api-tls\") pod \"maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp\" (UID: \"c13929af-4c3f-42f2-9179-716de50abacb\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp"
Sep 11 22:04:14.300146 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:14.300118    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp"
Sep 11 22:04:14.642405 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:14.642216    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp"]
Sep 11 22:04:14.794603 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:14.794567    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp" event={"ID":"c13929af-4c3f-42f2-9179-716de50abacb","Type":"ContainerStarted","Data":"9d9d742e38cf060cfa0d4619f338b78be4d0cea211f941f4165df7759b858902"}
Sep 11 22:04:14.902664 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:14.902579    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"]
Sep 11 22:04:14.907978 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:14.907948    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"
Sep 11 22:04:14.910759 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:14.910728    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f823e45b-serving-cert\""
Sep 11 22:04:14.919159 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:14.919122    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"]
Sep 11 22:04:14.996243 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:14.996191    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-openshift-service-ca\") pod \"maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns\" (UID: \"fcfbb5cc-7ce1-4800-9538-532266a0dc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"
Sep 11 22:04:14.996426 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:14.996250    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgnf7\" (UniqueName: \"kubernetes.io/projected/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-kube-api-access-sgnf7\") pod \"maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns\" (UID: \"fcfbb5cc-7ce1-4800-9538-532266a0dc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"
Sep 11 22:04:14.996426 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:14.996369    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-maas-api-tls\") pod \"maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns\" (UID: \"fcfbb5cc-7ce1-4800-9538-532266a0dc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"
Sep 11 22:04:14.996426 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:14.996413    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-metrics-certs\") pod \"maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns\" (UID: \"fcfbb5cc-7ce1-4800-9538-532266a0dc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"
Sep 11 22:04:15.097114 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:15.097087    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-maas-api-tls\") pod \"maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns\" (UID: \"fcfbb5cc-7ce1-4800-9538-532266a0dc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"
Sep 11 22:04:15.097192 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:15.097127    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-metrics-certs\") pod \"maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns\" (UID: \"fcfbb5cc-7ce1-4800-9538-532266a0dc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"
Sep 11 22:04:15.097242 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:15.097201    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-openshift-service-ca\") pod \"maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns\" (UID: \"fcfbb5cc-7ce1-4800-9538-532266a0dc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"
Sep 11 22:04:15.097242 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:15.097220    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgnf7\" (UniqueName: \"kubernetes.io/projected/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-kube-api-access-sgnf7\") pod \"maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns\" (UID: \"fcfbb5cc-7ce1-4800-9538-532266a0dc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"
Sep 11 22:04:15.097374 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:04:15.097255    2587 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-serving-cert: secret "maas-api-e2e-ait-f823e45b-serving-cert" not found
Sep 11 22:04:15.097374 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:04:15.097350    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-maas-api-tls podName:fcfbb5cc-7ce1-4800-9538-532266a0dc3c nodeName:}" failed. No retries permitted until 2026-09-11 22:04:15.597302888 +0000 UTC m=+708.772379246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-maas-api-tls") pod "maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns" (UID: "fcfbb5cc-7ce1-4800-9538-532266a0dc3c") : secret "maas-api-e2e-ait-f823e45b-serving-cert" not found
Sep 11 22:04:15.097891 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:15.097869    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-openshift-service-ca\") pod \"maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns\" (UID: \"fcfbb5cc-7ce1-4800-9538-532266a0dc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"
Sep 11 22:04:15.099647 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:15.099633    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-metrics-certs\") pod \"maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns\" (UID: \"fcfbb5cc-7ce1-4800-9538-532266a0dc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"
Sep 11 22:04:15.105339 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:15.105310    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgnf7\" (UniqueName: \"kubernetes.io/projected/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-kube-api-access-sgnf7\") pod \"maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns\" (UID: \"fcfbb5cc-7ce1-4800-9538-532266a0dc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"
Sep 11 22:04:15.602707 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:15.602667    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-maas-api-tls\") pod \"maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns\" (UID: \"fcfbb5cc-7ce1-4800-9538-532266a0dc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"
Sep 11 22:04:15.602944 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:04:15.602913    2587 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-serving-cert: secret "maas-api-e2e-ait-f823e45b-serving-cert" not found
Sep 11 22:04:15.603074 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:04:15.602992    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-maas-api-tls podName:fcfbb5cc-7ce1-4800-9538-532266a0dc3c nodeName:}" failed. No retries permitted until 2026-09-11 22:04:16.602972216 +0000 UTC m=+709.778048567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-maas-api-tls") pod "maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns" (UID: "fcfbb5cc-7ce1-4800-9538-532266a0dc3c") : secret "maas-api-e2e-ait-f823e45b-serving-cert" not found
Sep 11 22:04:15.803308 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:15.802839    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp" event={"ID":"c13929af-4c3f-42f2-9179-716de50abacb","Type":"ContainerStarted","Data":"0a2f451dd8b13cfb1b98a4ff07ace84d3f27e79e5d8937258023cf59dabb8df0"}
Sep 11 22:04:15.803308 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:15.803081    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp"
Sep 11 22:04:15.842461 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:15.842396    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp" podStartSLOduration=2.4247102099999998 podStartE2EDuration="2.842374205s" totalImagesPullingTime="417.663995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:15.837257948 +0000 UTC m=+709.012334347" watchObservedRunningTime="2026-09-11 22:04:15.842374205 +0000 UTC m=+709.017450576"
Sep 11 22:04:16.616016 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:16.615872    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-maas-api-tls\") pod \"maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns\" (UID: \"fcfbb5cc-7ce1-4800-9538-532266a0dc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"
Sep 11 22:04:16.619369 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:16.619295    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-maas-api-tls\") pod \"maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns\" (UID: \"fcfbb5cc-7ce1-4800-9538-532266a0dc3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"
Sep 11 22:04:16.741279 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:16.741236    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"
Sep 11 22:04:17.020071 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:17.019949    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"]
Sep 11 22:04:17.027632 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:04:17.025204    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcfbb5cc_7ce1_4800_9538_532266a0dc3c.slice/crio-70c083ea4472f5b67b70de49eed4ce1fa87e2f4da373a243cd7be79b0b038f62 WatchSource:0}: Error finding container 70c083ea4472f5b67b70de49eed4ce1fa87e2f4da373a243cd7be79b0b038f62: Status 404 returned error can't find the container with id 70c083ea4472f5b67b70de49eed4ce1fa87e2f4da373a243cd7be79b0b038f62
Sep 11 22:04:17.820650 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:17.820603    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns" event={"ID":"fcfbb5cc-7ce1-4800-9538-532266a0dc3c","Type":"ContainerStarted","Data":"f1ca3c99c4c6d8f3d4f418cb6b3a10c91a2fb5b7d3155cd1ca8f2e359edb9f79"}
Sep 11 22:04:17.820851 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:17.820667    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns" event={"ID":"fcfbb5cc-7ce1-4800-9538-532266a0dc3c","Type":"ContainerStarted","Data":"70c083ea4472f5b67b70de49eed4ce1fa87e2f4da373a243cd7be79b0b038f62"}
Sep 11 22:04:17.821502 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:17.821472    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"
Sep 11 22:04:18.087964 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:18.087842    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns" podStartSLOduration=3.72937603 podStartE2EDuration="4.087820197s" totalImagesPullingTime="358.444167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:17.839709379 +0000 UTC m=+711.014785749" watchObservedRunningTime="2026-09-11 22:04:18.087820197 +0000 UTC m=+711.262896568"
Sep 11 22:04:18.090358 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:18.090306    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:04:18.829279 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:18.829234    2587 generic.go:356] "Generic (PLEG): container finished" podID="9fe4daef-73cf-4488-acec-a2b20a88db5e" containerID="1d9b78d1272c6daf4a1dfb776ce66888821506805c8d7a20cf42c85144f0751b" exitCode=0
Sep 11 22:04:18.829543 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:18.829316    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk" event={"ID":"9fe4daef-73cf-4488-acec-a2b20a88db5e","Type":"ContainerDied","Data":"1d9b78d1272c6daf4a1dfb776ce66888821506805c8d7a20cf42c85144f0751b"}
Sep 11 22:04:18.829888 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:18.829869    2587 scope.go:117] "RemoveContainer" containerID="1d9b78d1272c6daf4a1dfb776ce66888821506805c8d7a20cf42c85144f0751b"
Sep 11 22:04:18.830956 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:18.830926    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:04:18.831074 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:18.830990    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:04:18.831074 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:18.831020    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:04:19.840871 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:19.840835    2587 generic.go:356] "Generic (PLEG): container finished" podID="f2f9ee75-e2ab-45e6-9f74-9088f1508251" containerID="d8c1f7b71fe6572df3397251cfd4a567d0ec3d9fabd779d5cae1031ff5b00699" exitCode=0
Sep 11 22:04:19.841625 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:19.841465    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq" event={"ID":"f2f9ee75-e2ab-45e6-9f74-9088f1508251","Type":"ContainerDied","Data":"d8c1f7b71fe6572df3397251cfd4a567d0ec3d9fabd779d5cae1031ff5b00699"}
Sep 11 22:04:19.842611 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:19.842592    2587 scope.go:117] "RemoveContainer" containerID="d8c1f7b71fe6572df3397251cfd4a567d0ec3d9fabd779d5cae1031ff5b00699"
Sep 11 22:04:19.844993 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:19.844184    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:04:19.844993 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:19.844273    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:04:19.844993 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:19.844362    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:04:19.863598 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:19.863401    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk" event={"ID":"9fe4daef-73cf-4488-acec-a2b20a88db5e","Type":"ContainerStarted","Data":"7d84bf3849690074d61886b832028800f52a0377d98d8b50543c6d4062db6a5a"}
Sep 11 22:04:20.470633 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:20.470581    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:04:20.470843 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:20.470820    2587 patch_prober.go:28] interesting pod/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 11 22:04:20.470923 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:20.470875    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk" podUID="9fe4daef-73cf-4488-acec-a2b20a88db5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 11 22:04:20.870923 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:20.870832    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq" event={"ID":"f2f9ee75-e2ab-45e6-9f74-9088f1508251","Type":"ContainerStarted","Data":"1063e304cdbd52beb4e3c869d51be44ceedf903bea53f7e83f14d679a293e932"}
Sep 11 22:04:21.316699 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:21.316659    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:04:21.316927 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:21.316899    2587 patch_prober.go:28] interesting pod/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 11 22:04:21.317012 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:21.316973    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq" podUID="f2f9ee75-e2ab-45e6-9f74-9088f1508251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 11 22:04:21.471081 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:21.471045    2587 patch_prober.go:28] interesting pod/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 11 22:04:21.471248 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:21.471113    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk" podUID="9fe4daef-73cf-4488-acec-a2b20a88db5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 11 22:04:21.818905 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:21.818875    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp"
Sep 11 22:04:22.317742 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:22.317711    2587 patch_prober.go:28] interesting pod/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 11 22:04:22.318201 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:22.318174    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq" podUID="f2f9ee75-e2ab-45e6-9f74-9088f1508251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 11 22:04:22.471831 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:22.471794    2587 patch_prober.go:28] interesting pod/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 11 22:04:22.472118 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:22.472090    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk" podUID="9fe4daef-73cf-4488-acec-a2b20a88db5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 11 22:04:23.317348 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:23.317299    2587 patch_prober.go:28] interesting pod/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 11 22:04:23.317518 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:23.317372    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq" podUID="f2f9ee75-e2ab-45e6-9f74-9088f1508251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 11 22:04:23.470798 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:23.470754    2587 patch_prober.go:28] interesting pod/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 11 22:04:23.471419 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:23.471385    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk" podUID="9fe4daef-73cf-4488-acec-a2b20a88db5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 11 22:04:24.318273 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:24.318245    2587 patch_prober.go:28] interesting pod/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 11 22:04:24.318498 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:24.318429    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq" podUID="f2f9ee75-e2ab-45e6-9f74-9088f1508251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 11 22:04:24.471139 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:24.471106    2587 patch_prober.go:28] interesting pod/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 11 22:04:24.471631 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:24.471599    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk" podUID="9fe4daef-73cf-4488-acec-a2b20a88db5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 11 22:04:24.595744 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:24.595660    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"]
Sep 11 22:04:24.600481 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:24.600458    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"
Sep 11 22:04:24.603531 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:24.603317    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-37182a-serving-cert\""
Sep 11 22:04:24.611062 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:24.611003    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"]
Sep 11 22:04:24.710539 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:24.710497    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9babb15f-7478-4134-aefc-9fefac8afd03-maas-api-tls\") pod \"maas-api-e2e-inf-b-37182a-757999cc59-6jh8j\" (UID: \"9babb15f-7478-4134-aefc-9fefac8afd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"
Sep 11 22:04:24.710739 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:24.710555    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9babb15f-7478-4134-aefc-9fefac8afd03-openshift-service-ca\") pod \"maas-api-e2e-inf-b-37182a-757999cc59-6jh8j\" (UID: \"9babb15f-7478-4134-aefc-9fefac8afd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"
Sep 11 22:04:24.710739 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:24.710624    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzf2g\" (UniqueName: \"kubernetes.io/projected/9babb15f-7478-4134-aefc-9fefac8afd03-kube-api-access-hzf2g\") pod \"maas-api-e2e-inf-b-37182a-757999cc59-6jh8j\" (UID: \"9babb15f-7478-4134-aefc-9fefac8afd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"
Sep 11 22:04:24.710739 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:24.710695    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9babb15f-7478-4134-aefc-9fefac8afd03-metrics-certs\") pod \"maas-api-e2e-inf-b-37182a-757999cc59-6jh8j\" (UID: \"9babb15f-7478-4134-aefc-9fefac8afd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"
Sep 11 22:04:24.812010 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:24.811975    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzf2g\" (UniqueName: \"kubernetes.io/projected/9babb15f-7478-4134-aefc-9fefac8afd03-kube-api-access-hzf2g\") pod \"maas-api-e2e-inf-b-37182a-757999cc59-6jh8j\" (UID: \"9babb15f-7478-4134-aefc-9fefac8afd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"
Sep 11 22:04:24.812189 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:24.812078    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9babb15f-7478-4134-aefc-9fefac8afd03-metrics-certs\") pod \"maas-api-e2e-inf-b-37182a-757999cc59-6jh8j\" (UID: \"9babb15f-7478-4134-aefc-9fefac8afd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"
Sep 11 22:04:24.812598 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:24.812580    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9babb15f-7478-4134-aefc-9fefac8afd03-maas-api-tls\") pod \"maas-api-e2e-inf-b-37182a-757999cc59-6jh8j\" (UID: \"9babb15f-7478-4134-aefc-9fefac8afd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"
Sep 11 22:04:24.812712 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:04:24.812696    2587 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-serving-cert: secret "maas-api-e2e-inf-b-37182a-serving-cert" not found
Sep 11 22:04:24.812769 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:04:24.812758    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9babb15f-7478-4134-aefc-9fefac8afd03-maas-api-tls podName:9babb15f-7478-4134-aefc-9fefac8afd03 nodeName:}" failed. No retries permitted until 2026-09-11 22:04:25.312739158 +0000 UTC m=+718.487815522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9babb15f-7478-4134-aefc-9fefac8afd03-maas-api-tls") pod "maas-api-e2e-inf-b-37182a-757999cc59-6jh8j" (UID: "9babb15f-7478-4134-aefc-9fefac8afd03") : secret "maas-api-e2e-inf-b-37182a-serving-cert" not found
Sep 11 22:04:24.813018 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:24.812998    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9babb15f-7478-4134-aefc-9fefac8afd03-openshift-service-ca\") pod \"maas-api-e2e-inf-b-37182a-757999cc59-6jh8j\" (UID: \"9babb15f-7478-4134-aefc-9fefac8afd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"
Sep 11 22:04:24.813341 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:24.813312    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9babb15f-7478-4134-aefc-9fefac8afd03-openshift-service-ca\") pod \"maas-api-e2e-inf-b-37182a-757999cc59-6jh8j\" (UID: \"9babb15f-7478-4134-aefc-9fefac8afd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"
Sep 11 22:04:24.816380 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:24.814449    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9babb15f-7478-4134-aefc-9fefac8afd03-metrics-certs\") pod \"maas-api-e2e-inf-b-37182a-757999cc59-6jh8j\" (UID: \"9babb15f-7478-4134-aefc-9fefac8afd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"
Sep 11 22:04:24.824130 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:24.824098    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzf2g\" (UniqueName: \"kubernetes.io/projected/9babb15f-7478-4134-aefc-9fefac8afd03-kube-api-access-hzf2g\") pod \"maas-api-e2e-inf-b-37182a-757999cc59-6jh8j\" (UID: \"9babb15f-7478-4134-aefc-9fefac8afd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"
Sep 11 22:04:24.879498 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:24.879016    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"
Sep 11 22:04:25.316849 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:25.316814    2587 patch_prober.go:28] interesting pod/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 11 22:04:25.317033 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:25.316882    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq" podUID="f2f9ee75-e2ab-45e6-9f74-9088f1508251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 11 22:04:25.317033 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:25.317015    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9babb15f-7478-4134-aefc-9fefac8afd03-maas-api-tls\") pod \"maas-api-e2e-inf-b-37182a-757999cc59-6jh8j\" (UID: \"9babb15f-7478-4134-aefc-9fefac8afd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"
Sep 11 22:04:25.317194 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:04:25.317171    2587 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-serving-cert: secret "maas-api-e2e-inf-b-37182a-serving-cert" not found
Sep 11 22:04:25.317255 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:04:25.317245    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9babb15f-7478-4134-aefc-9fefac8afd03-maas-api-tls podName:9babb15f-7478-4134-aefc-9fefac8afd03 nodeName:}" failed. No retries permitted until 2026-09-11 22:04:26.317224955 +0000 UTC m=+719.492301318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9babb15f-7478-4134-aefc-9fefac8afd03-maas-api-tls") pod "maas-api-e2e-inf-b-37182a-757999cc59-6jh8j" (UID: "9babb15f-7478-4134-aefc-9fefac8afd03") : secret "maas-api-e2e-inf-b-37182a-serving-cert" not found
Sep 11 22:04:25.585920 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:25.585832    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:04:25.586835 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:25.586817    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:04:25.587488 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:25.587466    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:04:26.317614 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:26.317574    2587 patch_prober.go:28] interesting pod/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 11 22:04:26.317814 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:26.317650    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq" podUID="f2f9ee75-e2ab-45e6-9f74-9088f1508251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 11 22:04:26.335079 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:26.335040    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9babb15f-7478-4134-aefc-9fefac8afd03-maas-api-tls\") pod \"maas-api-e2e-inf-b-37182a-757999cc59-6jh8j\" (UID: \"9babb15f-7478-4134-aefc-9fefac8afd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"
Sep 11 22:04:26.338538 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:26.338502    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9babb15f-7478-4134-aefc-9fefac8afd03-maas-api-tls\") pod \"maas-api-e2e-inf-b-37182a-757999cc59-6jh8j\" (UID: \"9babb15f-7478-4134-aefc-9fefac8afd03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"
Sep 11 22:04:26.416367 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:26.416301    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"
Sep 11 22:04:26.727352 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:26.724728    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"]
Sep 11 22:04:26.742675 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:04:26.732907    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9babb15f_7478_4134_aefc_9fefac8afd03.slice/crio-279ec09cebfb6ce3b666e7157b62027833a02368348c61df15adf037f6bcb38e WatchSource:0}: Error finding container 279ec09cebfb6ce3b666e7157b62027833a02368348c61df15adf037f6bcb38e: Status 404 returned error can't find the container with id 279ec09cebfb6ce3b666e7157b62027833a02368348c61df15adf037f6bcb38e
Sep 11 22:04:26.913245 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:26.913204    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j" event={"ID":"9babb15f-7478-4134-aefc-9fefac8afd03","Type":"ContainerStarted","Data":"279ec09cebfb6ce3b666e7157b62027833a02368348c61df15adf037f6bcb38e"}
Sep 11 22:04:27.326754 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:27.326719    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:04:27.327309 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:27.327287    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:04:27.328136 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:27.328116    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:04:27.921411 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:27.921372    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j" event={"ID":"9babb15f-7478-4134-aefc-9fefac8afd03","Type":"ContainerStarted","Data":"88981b5c9f5c397b51d0593221ec684d76285e7c61e706c355030152cce646b5"}
Sep 11 22:04:27.921852 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:27.921465    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"
Sep 11 22:04:27.939390 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:27.939301    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j" podStartSLOduration=3.549956241 podStartE2EDuration="3.939285992s" totalImagesPullingTime="389.329751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:27.938484227 +0000 UTC m=+721.113560601" watchObservedRunningTime="2026-09-11 22:04:27.939285992 +0000 UTC m=+721.114362363"
Sep 11 22:04:33.932578 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:33.932544    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"
Sep 11 22:04:36.676118 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:36.676080    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:04:50.962719 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:50.962685    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"]
Sep 11 22:04:50.963305 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:50.962947    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns" podUID="fcfbb5cc-7ce1-4800-9538-532266a0dc3c" containerName="maas-api" containerID="cri-o://f1ca3c99c4c6d8f3d4f418cb6b3a10c91a2fb5b7d3155cd1ca8f2e359edb9f79" gracePeriod=30
Sep 11 22:04:51.226570 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:51.226497    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"
Sep 11 22:04:51.308261 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:51.308227    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgnf7\" (UniqueName: \"kubernetes.io/projected/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-kube-api-access-sgnf7\") pod \"fcfbb5cc-7ce1-4800-9538-532266a0dc3c\" (UID: \"fcfbb5cc-7ce1-4800-9538-532266a0dc3c\") "
Sep 11 22:04:51.308507 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:51.308323    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-maas-api-tls\") pod \"fcfbb5cc-7ce1-4800-9538-532266a0dc3c\" (UID: \"fcfbb5cc-7ce1-4800-9538-532266a0dc3c\") "
Sep 11 22:04:51.308507 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:51.308398    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-metrics-certs\") pod \"fcfbb5cc-7ce1-4800-9538-532266a0dc3c\" (UID: \"fcfbb5cc-7ce1-4800-9538-532266a0dc3c\") "
Sep 11 22:04:51.308640 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:51.308513    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-openshift-service-ca\") pod \"fcfbb5cc-7ce1-4800-9538-532266a0dc3c\" (UID: \"fcfbb5cc-7ce1-4800-9538-532266a0dc3c\") "
Sep 11 22:04:51.309137 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:51.309109    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fcfbb5cc-7ce1-4800-9538-532266a0dc3c" (UID: "fcfbb5cc-7ce1-4800-9538-532266a0dc3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:04:51.310965 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:51.310918    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fcfbb5cc-7ce1-4800-9538-532266a0dc3c" (UID: "fcfbb5cc-7ce1-4800-9538-532266a0dc3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:04:51.312793 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:51.312762    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fcfbb5cc-7ce1-4800-9538-532266a0dc3c" (UID: "fcfbb5cc-7ce1-4800-9538-532266a0dc3c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:04:51.312953 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:51.312921    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-kube-api-access-sgnf7" (OuterVolumeSpecName: "kube-api-access-sgnf7") pod "fcfbb5cc-7ce1-4800-9538-532266a0dc3c" (UID: "fcfbb5cc-7ce1-4800-9538-532266a0dc3c"). InnerVolumeSpecName "kube-api-access-sgnf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:04:51.410314 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:51.409898    2587 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-openshift-service-ca\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:04:51.410314 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:51.409936    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgnf7\" (UniqueName: \"kubernetes.io/projected/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-kube-api-access-sgnf7\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:04:51.410314 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:51.409955    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-maas-api-tls\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:04:51.410314 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:51.409969    2587 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcfbb5cc-7ce1-4800-9538-532266a0dc3c-metrics-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:04:52.053537 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:52.053503    2587 generic.go:356] "Generic (PLEG): container finished" podID="fcfbb5cc-7ce1-4800-9538-532266a0dc3c" containerID="f1ca3c99c4c6d8f3d4f418cb6b3a10c91a2fb5b7d3155cd1ca8f2e359edb9f79" exitCode=0
Sep 11 22:04:52.053911 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:52.053550    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"
Sep 11 22:04:52.053911 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:52.053583    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns" event={"ID":"fcfbb5cc-7ce1-4800-9538-532266a0dc3c","Type":"ContainerDied","Data":"f1ca3c99c4c6d8f3d4f418cb6b3a10c91a2fb5b7d3155cd1ca8f2e359edb9f79"}
Sep 11 22:04:52.053911 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:52.053636    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns" event={"ID":"fcfbb5cc-7ce1-4800-9538-532266a0dc3c","Type":"ContainerDied","Data":"70c083ea4472f5b67b70de49eed4ce1fa87e2f4da373a243cd7be79b0b038f62"}
Sep 11 22:04:52.053911 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:52.053659    2587 scope.go:117] "RemoveContainer" containerID="f1ca3c99c4c6d8f3d4f418cb6b3a10c91a2fb5b7d3155cd1ca8f2e359edb9f79"
Sep 11 22:04:52.063657 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:52.063636    2587 scope.go:117] "RemoveContainer" containerID="f1ca3c99c4c6d8f3d4f418cb6b3a10c91a2fb5b7d3155cd1ca8f2e359edb9f79"
Sep 11 22:04:52.063971 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:04:52.063949    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1ca3c99c4c6d8f3d4f418cb6b3a10c91a2fb5b7d3155cd1ca8f2e359edb9f79\": container with ID starting with f1ca3c99c4c6d8f3d4f418cb6b3a10c91a2fb5b7d3155cd1ca8f2e359edb9f79 not found: ID does not exist" containerID="f1ca3c99c4c6d8f3d4f418cb6b3a10c91a2fb5b7d3155cd1ca8f2e359edb9f79"
Sep 11 22:04:52.064026 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:52.063980    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1ca3c99c4c6d8f3d4f418cb6b3a10c91a2fb5b7d3155cd1ca8f2e359edb9f79"} err="failed to get container status \"f1ca3c99c4c6d8f3d4f418cb6b3a10c91a2fb5b7d3155cd1ca8f2e359edb9f79\": rpc error: code = NotFound desc = could not find container \"f1ca3c99c4c6d8f3d4f418cb6b3a10c91a2fb5b7d3155cd1ca8f2e359edb9f79\": container with ID starting with f1ca3c99c4c6d8f3d4f418cb6b3a10c91a2fb5b7d3155cd1ca8f2e359edb9f79 not found: ID does not exist"
Sep 11 22:04:52.076929 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:52.076899    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"]
Sep 11 22:04:52.079181 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:52.079159    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f823e45b-85899dcdf8-xbpns"]
Sep 11 22:04:53.336303 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:53.336258    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcfbb5cc-7ce1-4800-9538-532266a0dc3c" path="/var/lib/kubelet/pods/fcfbb5cc-7ce1-4800-9538-532266a0dc3c/volumes"
Sep 11 22:04:56.370562 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:56.370528    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd"]
Sep 11 22:04:56.371158 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:56.371142    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcfbb5cc-7ce1-4800-9538-532266a0dc3c" containerName="maas-api"
Sep 11 22:04:56.371233 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:56.371160    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcfbb5cc-7ce1-4800-9538-532266a0dc3c" containerName="maas-api"
Sep 11 22:04:56.371272 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:56.371265    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcfbb5cc-7ce1-4800-9538-532266a0dc3c" containerName="maas-api"
Sep 11 22:04:56.375274 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:56.375244    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd"
Sep 11 22:04:56.377815 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:56.377792    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-0cd94651-serving-cert\""
Sep 11 22:04:56.387163 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:56.387138    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd"]
Sep 11 22:04:56.460661 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:56.460621    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6015baa9-8ca3-432b-b915-286c51934f44-maas-api-tls\") pod \"maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd\" (UID: \"6015baa9-8ca3-432b-b915-286c51934f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd"
Sep 11 22:04:56.460661 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:56.460658    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfp48\" (UniqueName: \"kubernetes.io/projected/6015baa9-8ca3-432b-b915-286c51934f44-kube-api-access-zfp48\") pod \"maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd\" (UID: \"6015baa9-8ca3-432b-b915-286c51934f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd"
Sep 11 22:04:56.460864 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:56.460785    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6015baa9-8ca3-432b-b915-286c51934f44-metrics-certs\") pod \"maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd\" (UID: \"6015baa9-8ca3-432b-b915-286c51934f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd"
Sep 11 22:04:56.460864 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:56.460843    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6015baa9-8ca3-432b-b915-286c51934f44-openshift-service-ca\") pod \"maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd\" (UID: \"6015baa9-8ca3-432b-b915-286c51934f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd"
Sep 11 22:04:56.561811 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:56.561771    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6015baa9-8ca3-432b-b915-286c51934f44-maas-api-tls\") pod \"maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd\" (UID: \"6015baa9-8ca3-432b-b915-286c51934f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd"
Sep 11 22:04:56.561811 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:56.561809    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfp48\" (UniqueName: \"kubernetes.io/projected/6015baa9-8ca3-432b-b915-286c51934f44-kube-api-access-zfp48\") pod \"maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd\" (UID: \"6015baa9-8ca3-432b-b915-286c51934f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd"
Sep 11 22:04:56.562061 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:56.561870    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6015baa9-8ca3-432b-b915-286c51934f44-metrics-certs\") pod \"maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd\" (UID: \"6015baa9-8ca3-432b-b915-286c51934f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd"
Sep 11 22:04:56.562061 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:56.561915    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6015baa9-8ca3-432b-b915-286c51934f44-openshift-service-ca\") pod \"maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd\" (UID: \"6015baa9-8ca3-432b-b915-286c51934f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd"
Sep 11 22:04:56.562633 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:56.562611    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6015baa9-8ca3-432b-b915-286c51934f44-openshift-service-ca\") pod \"maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd\" (UID: \"6015baa9-8ca3-432b-b915-286c51934f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd"
Sep 11 22:04:56.564370 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:56.564340    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6015baa9-8ca3-432b-b915-286c51934f44-maas-api-tls\") pod \"maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd\" (UID: \"6015baa9-8ca3-432b-b915-286c51934f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd"
Sep 11 22:04:56.564467 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:56.564323    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6015baa9-8ca3-432b-b915-286c51934f44-metrics-certs\") pod \"maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd\" (UID: \"6015baa9-8ca3-432b-b915-286c51934f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd"
Sep 11 22:04:56.571181 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:56.571158    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfp48\" (UniqueName: \"kubernetes.io/projected/6015baa9-8ca3-432b-b915-286c51934f44-kube-api-access-zfp48\") pod \"maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd\" (UID: \"6015baa9-8ca3-432b-b915-286c51934f44\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd"
Sep 11 22:04:56.686792 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:56.686754    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd"
Sep 11 22:04:56.883563 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:56.883519    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:04:56.977937 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:56.976452    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd"]
Sep 11 22:04:56.984728 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:04:56.983960    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6015baa9_8ca3_432b_b915_286c51934f44.slice/crio-208f8407965b7208691ae7cac90b765c25566a9e42366f0c4ee2b90eb4a7ef94 WatchSource:0}: Error finding container 208f8407965b7208691ae7cac90b765c25566a9e42366f0c4ee2b90eb4a7ef94: Status 404 returned error can't find the container with id 208f8407965b7208691ae7cac90b765c25566a9e42366f0c4ee2b90eb4a7ef94
Sep 11 22:04:57.084061 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:57.084020    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd" event={"ID":"6015baa9-8ca3-432b-b915-286c51934f44","Type":"ContainerStarted","Data":"208f8407965b7208691ae7cac90b765c25566a9e42366f0c4ee2b90eb4a7ef94"}
Sep 11 22:04:58.090839 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:58.090804    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd" event={"ID":"6015baa9-8ca3-432b-b915-286c51934f44","Type":"ContainerStarted","Data":"a97dbd24058ded5acd557a934a8013b95ff48b5f8edd6ea25a80f2efe6137d0f"}
Sep 11 22:04:58.091252 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:58.090925    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd"
Sep 11 22:04:58.109075 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:04:58.109019    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd" podStartSLOduration=1.712101415 podStartE2EDuration="2.109002554s" totalImagesPullingTime="396.901139ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:04:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:04:58.106642562 +0000 UTC m=+751.281718950" watchObservedRunningTime="2026-09-11 22:04:58.109002554 +0000 UTC m=+751.284078924"
Sep 11 22:05:04.102607 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:04.102578    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd"
Sep 11 22:05:10.576727 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:10.576686    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:05:24.682700 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:24.682659    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:05:26.928487 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:26.928384    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd"]
Sep 11 22:05:26.928970 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:26.928639    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd" podUID="6015baa9-8ca3-432b-b915-286c51934f44" containerName="maas-api" containerID="cri-o://a97dbd24058ded5acd557a934a8013b95ff48b5f8edd6ea25a80f2efe6137d0f" gracePeriod=30
Sep 11 22:05:27.191169 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:27.191100    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd"
Sep 11 22:05:27.262071 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:27.261351    2587 generic.go:356] "Generic (PLEG): container finished" podID="6015baa9-8ca3-432b-b915-286c51934f44" containerID="a97dbd24058ded5acd557a934a8013b95ff48b5f8edd6ea25a80f2efe6137d0f" exitCode=0
Sep 11 22:05:27.262071 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:27.261442    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd" event={"ID":"6015baa9-8ca3-432b-b915-286c51934f44","Type":"ContainerDied","Data":"a97dbd24058ded5acd557a934a8013b95ff48b5f8edd6ea25a80f2efe6137d0f"}
Sep 11 22:05:27.262071 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:27.261537    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd" event={"ID":"6015baa9-8ca3-432b-b915-286c51934f44","Type":"ContainerDied","Data":"208f8407965b7208691ae7cac90b765c25566a9e42366f0c4ee2b90eb4a7ef94"}
Sep 11 22:05:27.262071 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:27.261572    2587 scope.go:117] "RemoveContainer" containerID="a97dbd24058ded5acd557a934a8013b95ff48b5f8edd6ea25a80f2efe6137d0f"
Sep 11 22:05:27.262071 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:27.261741    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd"
Sep 11 22:05:27.275980 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:27.275955    2587 scope.go:117] "RemoveContainer" containerID="a97dbd24058ded5acd557a934a8013b95ff48b5f8edd6ea25a80f2efe6137d0f"
Sep 11 22:05:27.276162 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:27.276144    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6015baa9-8ca3-432b-b915-286c51934f44-metrics-certs\") pod \"6015baa9-8ca3-432b-b915-286c51934f44\" (UID: \"6015baa9-8ca3-432b-b915-286c51934f44\") "
Sep 11 22:05:27.276252 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:27.276239    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfp48\" (UniqueName: \"kubernetes.io/projected/6015baa9-8ca3-432b-b915-286c51934f44-kube-api-access-zfp48\") pod \"6015baa9-8ca3-432b-b915-286c51934f44\" (UID: \"6015baa9-8ca3-432b-b915-286c51934f44\") "
Sep 11 22:05:27.276314 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:27.276277    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6015baa9-8ca3-432b-b915-286c51934f44-openshift-service-ca\") pod \"6015baa9-8ca3-432b-b915-286c51934f44\" (UID: \"6015baa9-8ca3-432b-b915-286c51934f44\") "
Sep 11 22:05:27.276314 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:27.276295    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6015baa9-8ca3-432b-b915-286c51934f44-maas-api-tls\") pod \"6015baa9-8ca3-432b-b915-286c51934f44\" (UID: \"6015baa9-8ca3-432b-b915-286c51934f44\") "
Sep 11 22:05:27.277118 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:27.277058    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6015baa9-8ca3-432b-b915-286c51934f44-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6015baa9-8ca3-432b-b915-286c51934f44" (UID: "6015baa9-8ca3-432b-b915-286c51934f44"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:05:27.277570 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:05:27.277487    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a97dbd24058ded5acd557a934a8013b95ff48b5f8edd6ea25a80f2efe6137d0f\": container with ID starting with a97dbd24058ded5acd557a934a8013b95ff48b5f8edd6ea25a80f2efe6137d0f not found: ID does not exist" containerID="a97dbd24058ded5acd557a934a8013b95ff48b5f8edd6ea25a80f2efe6137d0f"
Sep 11 22:05:27.277570 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:27.277528    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a97dbd24058ded5acd557a934a8013b95ff48b5f8edd6ea25a80f2efe6137d0f"} err="failed to get container status \"a97dbd24058ded5acd557a934a8013b95ff48b5f8edd6ea25a80f2efe6137d0f\": rpc error: code = NotFound desc = could not find container \"a97dbd24058ded5acd557a934a8013b95ff48b5f8edd6ea25a80f2efe6137d0f\": container with ID starting with a97dbd24058ded5acd557a934a8013b95ff48b5f8edd6ea25a80f2efe6137d0f not found: ID does not exist"
Sep 11 22:05:27.279390 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:27.279363    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6015baa9-8ca3-432b-b915-286c51934f44-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6015baa9-8ca3-432b-b915-286c51934f44" (UID: "6015baa9-8ca3-432b-b915-286c51934f44"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:05:27.279604 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:27.279578    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6015baa9-8ca3-432b-b915-286c51934f44-kube-api-access-zfp48" (OuterVolumeSpecName: "kube-api-access-zfp48") pod "6015baa9-8ca3-432b-b915-286c51934f44" (UID: "6015baa9-8ca3-432b-b915-286c51934f44"). InnerVolumeSpecName "kube-api-access-zfp48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:05:27.281643 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:27.281617    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6015baa9-8ca3-432b-b915-286c51934f44-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6015baa9-8ca3-432b-b915-286c51934f44" (UID: "6015baa9-8ca3-432b-b915-286c51934f44"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:05:27.378018 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:27.377982    2587 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6015baa9-8ca3-432b-b915-286c51934f44-openshift-service-ca\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:05:27.378018 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:27.378017    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6015baa9-8ca3-432b-b915-286c51934f44-maas-api-tls\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:05:27.378272 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:27.378030    2587 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6015baa9-8ca3-432b-b915-286c51934f44-metrics-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:05:27.378272 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:27.378042    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfp48\" (UniqueName: \"kubernetes.io/projected/6015baa9-8ca3-432b-b915-286c51934f44-kube-api-access-zfp48\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:05:27.580701 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:27.580622    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd"]
Sep 11 22:05:27.584783 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:27.584755    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0cd94651-5c9cffcd7d-55lhd"]
Sep 11 22:05:28.044300 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.044259    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"]
Sep 11 22:05:28.044987 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.044957    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6015baa9-8ca3-432b-b915-286c51934f44" containerName="maas-api"
Sep 11 22:05:28.044987 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.044988    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="6015baa9-8ca3-432b-b915-286c51934f44" containerName="maas-api"
Sep 11 22:05:28.045138 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.045131    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="6015baa9-8ca3-432b-b915-286c51934f44" containerName="maas-api"
Sep 11 22:05:28.050037 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.050016    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.053014 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.052990    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6f399ea1-openshift-default-dockercfg-cldvw\""
Sep 11 22:05:28.061241 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.061218    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"]
Sep 11 22:05:28.085221 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.085180    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8244\" (UniqueName: \"kubernetes.io/projected/f678d2a9-5641-400a-8fac-8b1a026eb82a-kube-api-access-r8244\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.085428 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.085238    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f678d2a9-5641-400a-8fac-8b1a026eb82a-istio-token\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.085428 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.085276    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-workload-certs\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.085428 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.085308    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f678d2a9-5641-400a-8fac-8b1a026eb82a-istio-podinfo\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.085428 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.085357    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-workload-socket\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.085428 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.085417    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-istio-data\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.085739 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.085450    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-istio-envoy\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.085739 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.085535    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-credential-socket\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.085739 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.085574    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f678d2a9-5641-400a-8fac-8b1a026eb82a-istiod-ca-cert\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.186468 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.186430    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-credential-socket\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.186648 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.186495    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f678d2a9-5641-400a-8fac-8b1a026eb82a-istiod-ca-cert\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.186648 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.186545    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8244\" (UniqueName: \"kubernetes.io/projected/f678d2a9-5641-400a-8fac-8b1a026eb82a-kube-api-access-r8244\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.186648 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.186580    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f678d2a9-5641-400a-8fac-8b1a026eb82a-istio-token\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.186648 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.186617    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-workload-certs\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.186909 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.186650    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f678d2a9-5641-400a-8fac-8b1a026eb82a-istio-podinfo\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.186909 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.186686    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-workload-socket\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.186909 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.186715    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-istio-data\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.186909 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.186740    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-istio-envoy\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.187120 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.186903    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-credential-socket\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.187120 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.186975    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-workload-certs\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.187120 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.187040    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-workload-socket\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.187276 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.187237    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-istio-data\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.187663 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.187637    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f678d2a9-5641-400a-8fac-8b1a026eb82a-istiod-ca-cert\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.189203 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.189175    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-istio-envoy\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.189291 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.189262    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f678d2a9-5641-400a-8fac-8b1a026eb82a-istio-podinfo\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.195121 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.195100    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f678d2a9-5641-400a-8fac-8b1a026eb82a-istio-token\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.195360 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.195339    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8244\" (UniqueName: \"kubernetes.io/projected/f678d2a9-5641-400a-8fac-8b1a026eb82a-kube-api-access-r8244\") pod \"e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") " pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.363339 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.363234    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:28.645060 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.645026    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"]
Sep 11 22:05:28.646418 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:05:28.646392    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf678d2a9_5641_400a_8fac_8b1a026eb82a.slice/crio-5188e185d7cfff192a23e6ab4774376ac97673865ce29e1d9731bfbd97d65ca3 WatchSource:0}: Error finding container 5188e185d7cfff192a23e6ab4774376ac97673865ce29e1d9731bfbd97d65ca3: Status 404 returned error can't find the container with id 5188e185d7cfff192a23e6ab4774376ac97673865ce29e1d9731bfbd97d65ca3
Sep 11 22:05:28.648989 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.648957    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:05:28.649082 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.649040    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:05:28.649221 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:28.649086    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:05:29.275048 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:29.275011    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff" event={"ID":"f678d2a9-5641-400a-8fac-8b1a026eb82a","Type":"ContainerStarted","Data":"a262eb0008478ee6c48b4c2eb8601f615e7d8fa47ed90ced8fa4de7aa7fb6a4c"}
Sep 11 22:05:29.275558 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:29.275065    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff" event={"ID":"f678d2a9-5641-400a-8fac-8b1a026eb82a","Type":"ContainerStarted","Data":"5188e185d7cfff192a23e6ab4774376ac97673865ce29e1d9731bfbd97d65ca3"}
Sep 11 22:05:29.297004 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:29.296946    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff" podStartSLOduration=1.296928884 podStartE2EDuration="1.296928884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:05:29.295711715 +0000 UTC m=+782.470788085" watchObservedRunningTime="2026-09-11 22:05:29.296928884 +0000 UTC m=+782.472005254"
Sep 11 22:05:29.336974 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:29.336939    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6015baa9-8ca3-432b-b915-286c51934f44" path="/var/lib/kubelet/pods/6015baa9-8ca3-432b-b915-286c51934f44/volumes"
Sep 11 22:05:29.363827 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:29.363792    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:29.369058 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:29.369030    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:30.280845 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:30.280807    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:30.281922 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:30.281901    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:44.275892 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:44.275855    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:05:55.423417 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:55.423378    2587 generic.go:356] "Generic (PLEG): container finished" podID="f678d2a9-5641-400a-8fac-8b1a026eb82a" containerID="a262eb0008478ee6c48b4c2eb8601f615e7d8fa47ed90ced8fa4de7aa7fb6a4c" exitCode=0
Sep 11 22:05:55.423952 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:55.423459    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff" event={"ID":"f678d2a9-5641-400a-8fac-8b1a026eb82a","Type":"ContainerDied","Data":"a262eb0008478ee6c48b4c2eb8601f615e7d8fa47ed90ced8fa4de7aa7fb6a4c"}
Sep 11 22:05:55.424052 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:55.424034    2587 scope.go:117] "RemoveContainer" containerID="a262eb0008478ee6c48b4c2eb8601f615e7d8fa47ed90ced8fa4de7aa7fb6a4c"
Sep 11 22:05:55.425411 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:55.425382    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:05:55.425528 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:55.425445    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:05:55.425528 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:55.425481    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:05:56.441911 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:56.441877    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff" event={"ID":"f678d2a9-5641-400a-8fac-8b1a026eb82a","Type":"ContainerStarted","Data":"82ad9d9b0abcfcdadba629671175492031f7ab6b67a406f49eca21ba01e28b6d"}
Sep 11 22:05:57.364069 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:57.364031    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:05:57.365048 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:57.365024    2587 patch_prober.go:28] interesting pod/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 11 22:05:57.365164 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:57.365070    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff" podUID="f678d2a9-5641-400a-8fac-8b1a026eb82a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 11 22:05:58.364480 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:58.364437    2587 patch_prober.go:28] interesting pod/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 11 22:05:58.364877 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:58.364514    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff" podUID="f678d2a9-5641-400a-8fac-8b1a026eb82a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 11 22:05:58.996202 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:58.996164    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:05:59.364068 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:59.363963    2587 patch_prober.go:28] interesting pod/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 11 22:05:59.364068 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:05:59.364026    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff" podUID="f678d2a9-5641-400a-8fac-8b1a026eb82a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 11 22:06:00.281547 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:00.281514    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:06:00.363773 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:00.363730    2587 patch_prober.go:28] interesting pod/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 11 22:06:00.363942 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:00.363818    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff" podUID="f678d2a9-5641-400a-8fac-8b1a026eb82a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 11 22:06:01.364053 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:01.364017    2587 patch_prober.go:28] interesting pod/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Sep 11 22:06:01.364462 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:01.364082    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff" podUID="f678d2a9-5641-400a-8fac-8b1a026eb82a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Sep 11 22:06:02.368459 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:02.368427    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:06:02.369563 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:02.369545    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:06:14.273853 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:14.273817    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"]
Sep 11 22:06:14.274352 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:14.274041    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff" podUID="f678d2a9-5641-400a-8fac-8b1a026eb82a" containerName="istio-proxy" containerID="cri-o://82ad9d9b0abcfcdadba629671175492031f7ab6b67a406f49eca21ba01e28b6d" gracePeriod=30
Sep 11 22:06:15.046260 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.046225    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"]
Sep 11 22:06:15.050545 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.050517    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.053609 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.053582    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-2c9db036-openshift-default-dockercfg-5dst7\""
Sep 11 22:06:15.063784 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.063736    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"]
Sep 11 22:06:15.174637 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.174597    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8030325-9dfe-4d58-9d37-5baba7ceee45-istiod-ca-cert\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.174840 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.174699    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-istio-data\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.174840 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.174739    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8030325-9dfe-4d58-9d37-5baba7ceee45-istio-podinfo\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.174840 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.174781    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-workload-socket\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.174840 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.174808    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdzdd\" (UniqueName: \"kubernetes.io/projected/f8030325-9dfe-4d58-9d37-5baba7ceee45-kube-api-access-fdzdd\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.175033 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.174884    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-istio-envoy\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.175033 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.174967    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-credential-socket\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.175113 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.175045    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-workload-certs\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.175113 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.175077    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8030325-9dfe-4d58-9d37-5baba7ceee45-istio-token\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.276456 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.276420    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-istio-data\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.277086 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.276469    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8030325-9dfe-4d58-9d37-5baba7ceee45-istio-podinfo\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.277086 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.276493    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-workload-socket\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.277086 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.276525    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdzdd\" (UniqueName: \"kubernetes.io/projected/f8030325-9dfe-4d58-9d37-5baba7ceee45-kube-api-access-fdzdd\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.277086 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.276552    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-istio-envoy\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.277086 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.276595    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-credential-socket\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.277086 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.276674    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-workload-certs\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.277086 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.276714    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8030325-9dfe-4d58-9d37-5baba7ceee45-istio-token\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.277086 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.276754    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8030325-9dfe-4d58-9d37-5baba7ceee45-istiod-ca-cert\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.277086 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.276922    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-istio-data\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.277086 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.277016    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-credential-socket\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.277669 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.277140    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-workload-certs\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.277669 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.277265    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-workload-socket\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.277669 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.277516    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8030325-9dfe-4d58-9d37-5baba7ceee45-istiod-ca-cert\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.278900 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.278875    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-istio-envoy\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.279291 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.279265    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8030325-9dfe-4d58-9d37-5baba7ceee45-istio-podinfo\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.284465 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.284446    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8030325-9dfe-4d58-9d37-5baba7ceee45-istio-token\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.284702 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.284685    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdzdd\" (UniqueName: \"kubernetes.io/projected/f8030325-9dfe-4d58-9d37-5baba7ceee45-kube-api-access-fdzdd\") pod \"e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") " pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.363155 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.363048    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:15.646899 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.646868    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"]
Sep 11 22:06:15.648893 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:06:15.648865    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8030325_9dfe_4d58_9d37_5baba7ceee45.slice/crio-3f1577f7bf07e71234f360400d03514cad83079d8ae3ae0af10a9e1fb989d82b WatchSource:0}: Error finding container 3f1577f7bf07e71234f360400d03514cad83079d8ae3ae0af10a9e1fb989d82b: Status 404 returned error can't find the container with id 3f1577f7bf07e71234f360400d03514cad83079d8ae3ae0af10a9e1fb989d82b
Sep 11 22:06:15.651533 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.651493    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:06:15.651812 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.651786    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:06:15.651882 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:15.651835    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:06:16.563683 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:16.563640    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q" event={"ID":"f8030325-9dfe-4d58-9d37-5baba7ceee45","Type":"ContainerStarted","Data":"4f856e97e9e9822d0693e1b6518c1efe7f1af1ef4d72ab9e3e73672fa8f29d9f"}
Sep 11 22:06:16.564068 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:16.563703    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q" event={"ID":"f8030325-9dfe-4d58-9d37-5baba7ceee45","Type":"ContainerStarted","Data":"3f1577f7bf07e71234f360400d03514cad83079d8ae3ae0af10a9e1fb989d82b"}
Sep 11 22:06:16.595121 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:16.595068    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q" podStartSLOduration=1.595051795 podStartE2EDuration="1.595051795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:06:16.591925281 +0000 UTC m=+829.767001674" watchObservedRunningTime="2026-09-11 22:06:16.595051795 +0000 UTC m=+829.770128164"
Sep 11 22:06:17.363618 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:17.363586    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:17.368463 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:17.368437    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:17.370176 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:17.370147    2587 patch_prober.go:28] interesting pod/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 22:06:17.370281 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:17.370217    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff" podUID="f678d2a9-5641-400a-8fac-8b1a026eb82a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 22:06:17.569415 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:17.569372    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:17.570290 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:17.570273    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:19.526945 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.526917    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:06:19.584005 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.583917    2587 generic.go:356] "Generic (PLEG): container finished" podID="f678d2a9-5641-400a-8fac-8b1a026eb82a" containerID="82ad9d9b0abcfcdadba629671175492031f7ab6b67a406f49eca21ba01e28b6d" exitCode=0
Sep 11 22:06:19.584005 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.583984    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"
Sep 11 22:06:19.584209 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.584008    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff" event={"ID":"f678d2a9-5641-400a-8fac-8b1a026eb82a","Type":"ContainerDied","Data":"82ad9d9b0abcfcdadba629671175492031f7ab6b67a406f49eca21ba01e28b6d"}
Sep 11 22:06:19.584209 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.584061    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff" event={"ID":"f678d2a9-5641-400a-8fac-8b1a026eb82a","Type":"ContainerDied","Data":"5188e185d7cfff192a23e6ab4774376ac97673865ce29e1d9731bfbd97d65ca3"}
Sep 11 22:06:19.584209 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.584080    2587 scope.go:117] "RemoveContainer" containerID="82ad9d9b0abcfcdadba629671175492031f7ab6b67a406f49eca21ba01e28b6d"
Sep 11 22:06:19.595913 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.595890    2587 scope.go:117] "RemoveContainer" containerID="a262eb0008478ee6c48b4c2eb8601f615e7d8fa47ed90ced8fa4de7aa7fb6a4c"
Sep 11 22:06:19.604226 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.604199    2587 scope.go:117] "RemoveContainer" containerID="82ad9d9b0abcfcdadba629671175492031f7ab6b67a406f49eca21ba01e28b6d"
Sep 11 22:06:19.604574 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:06:19.604550    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82ad9d9b0abcfcdadba629671175492031f7ab6b67a406f49eca21ba01e28b6d\": container with ID starting with 82ad9d9b0abcfcdadba629671175492031f7ab6b67a406f49eca21ba01e28b6d not found: ID does not exist" containerID="82ad9d9b0abcfcdadba629671175492031f7ab6b67a406f49eca21ba01e28b6d"
Sep 11 22:06:19.604672 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.604581    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82ad9d9b0abcfcdadba629671175492031f7ab6b67a406f49eca21ba01e28b6d"} err="failed to get container status \"82ad9d9b0abcfcdadba629671175492031f7ab6b67a406f49eca21ba01e28b6d\": rpc error: code = NotFound desc = could not find container \"82ad9d9b0abcfcdadba629671175492031f7ab6b67a406f49eca21ba01e28b6d\": container with ID starting with 82ad9d9b0abcfcdadba629671175492031f7ab6b67a406f49eca21ba01e28b6d not found: ID does not exist"
Sep 11 22:06:19.604672 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.604598    2587 scope.go:117] "RemoveContainer" containerID="a262eb0008478ee6c48b4c2eb8601f615e7d8fa47ed90ced8fa4de7aa7fb6a4c"
Sep 11 22:06:19.604882 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:06:19.604860    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a262eb0008478ee6c48b4c2eb8601f615e7d8fa47ed90ced8fa4de7aa7fb6a4c\": container with ID starting with a262eb0008478ee6c48b4c2eb8601f615e7d8fa47ed90ced8fa4de7aa7fb6a4c not found: ID does not exist" containerID="a262eb0008478ee6c48b4c2eb8601f615e7d8fa47ed90ced8fa4de7aa7fb6a4c"
Sep 11 22:06:19.604932 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.604891    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a262eb0008478ee6c48b4c2eb8601f615e7d8fa47ed90ced8fa4de7aa7fb6a4c"} err="failed to get container status \"a262eb0008478ee6c48b4c2eb8601f615e7d8fa47ed90ced8fa4de7aa7fb6a4c\": rpc error: code = NotFound desc = could not find container \"a262eb0008478ee6c48b4c2eb8601f615e7d8fa47ed90ced8fa4de7aa7fb6a4c\": container with ID starting with a262eb0008478ee6c48b4c2eb8601f615e7d8fa47ed90ced8fa4de7aa7fb6a4c not found: ID does not exist"
Sep 11 22:06:19.630260 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.630232    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-workload-socket\") pod \"f678d2a9-5641-400a-8fac-8b1a026eb82a\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") "
Sep 11 22:06:19.630410 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.630340    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f678d2a9-5641-400a-8fac-8b1a026eb82a-istiod-ca-cert\") pod \"f678d2a9-5641-400a-8fac-8b1a026eb82a\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") "
Sep 11 22:06:19.630410 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.630368    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-istio-data\") pod \"f678d2a9-5641-400a-8fac-8b1a026eb82a\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") "
Sep 11 22:06:19.630410 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.630396    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-credential-socket\") pod \"f678d2a9-5641-400a-8fac-8b1a026eb82a\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") "
Sep 11 22:06:19.630591 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.630458    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f678d2a9-5641-400a-8fac-8b1a026eb82a-istio-podinfo\") pod \"f678d2a9-5641-400a-8fac-8b1a026eb82a\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") "
Sep 11 22:06:19.630591 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.630507    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8244\" (UniqueName: \"kubernetes.io/projected/f678d2a9-5641-400a-8fac-8b1a026eb82a-kube-api-access-r8244\") pod \"f678d2a9-5641-400a-8fac-8b1a026eb82a\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") "
Sep 11 22:06:19.630591 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.630520    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f678d2a9-5641-400a-8fac-8b1a026eb82a" (UID: "f678d2a9-5641-400a-8fac-8b1a026eb82a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:06:19.630591 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.630531    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-istio-envoy\") pod \"f678d2a9-5641-400a-8fac-8b1a026eb82a\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") "
Sep 11 22:06:19.630591 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.630565    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f678d2a9-5641-400a-8fac-8b1a026eb82a-istio-token\") pod \"f678d2a9-5641-400a-8fac-8b1a026eb82a\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") "
Sep 11 22:06:19.630822 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.630620    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-istio-data" (OuterVolumeSpecName: "istio-data") pod "f678d2a9-5641-400a-8fac-8b1a026eb82a" (UID: "f678d2a9-5641-400a-8fac-8b1a026eb82a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:06:19.630822 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.630619    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-workload-certs\") pod \"f678d2a9-5641-400a-8fac-8b1a026eb82a\" (UID: \"f678d2a9-5641-400a-8fac-8b1a026eb82a\") "
Sep 11 22:06:19.630822 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.630792    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f678d2a9-5641-400a-8fac-8b1a026eb82a" (UID: "f678d2a9-5641-400a-8fac-8b1a026eb82a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:06:19.630985 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.630834    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f678d2a9-5641-400a-8fac-8b1a026eb82a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f678d2a9-5641-400a-8fac-8b1a026eb82a" (UID: "f678d2a9-5641-400a-8fac-8b1a026eb82a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:06:19.631223 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.631133    2587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-workload-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:06:19.631223 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.631149    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f678d2a9-5641-400a-8fac-8b1a026eb82a" (UID: "f678d2a9-5641-400a-8fac-8b1a026eb82a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:06:19.631223 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.631171    2587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f678d2a9-5641-400a-8fac-8b1a026eb82a-istiod-ca-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:06:19.631223 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.631186    2587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-istio-data\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:06:19.631223 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.631199    2587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-credential-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:06:19.632757 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.632732    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f678d2a9-5641-400a-8fac-8b1a026eb82a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f678d2a9-5641-400a-8fac-8b1a026eb82a" (UID: "f678d2a9-5641-400a-8fac-8b1a026eb82a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:06:19.633232 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.633210    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f678d2a9-5641-400a-8fac-8b1a026eb82a-istio-token" (OuterVolumeSpecName: "istio-token") pod "f678d2a9-5641-400a-8fac-8b1a026eb82a" (UID: "f678d2a9-5641-400a-8fac-8b1a026eb82a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:06:19.633347 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.633267    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f678d2a9-5641-400a-8fac-8b1a026eb82a" (UID: "f678d2a9-5641-400a-8fac-8b1a026eb82a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:06:19.633513 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.633487    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f678d2a9-5641-400a-8fac-8b1a026eb82a-kube-api-access-r8244" (OuterVolumeSpecName: "kube-api-access-r8244") pod "f678d2a9-5641-400a-8fac-8b1a026eb82a" (UID: "f678d2a9-5641-400a-8fac-8b1a026eb82a"). InnerVolumeSpecName "kube-api-access-r8244". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:06:19.732144 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.732093    2587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f678d2a9-5641-400a-8fac-8b1a026eb82a-istio-podinfo\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:06:19.732144 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.732120    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8244\" (UniqueName: \"kubernetes.io/projected/f678d2a9-5641-400a-8fac-8b1a026eb82a-kube-api-access-r8244\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:06:19.732144 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.732135    2587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-istio-envoy\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:06:19.732144 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.732148    2587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f678d2a9-5641-400a-8fac-8b1a026eb82a-istio-token\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:06:19.732517 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.732159    2587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f678d2a9-5641-400a-8fac-8b1a026eb82a-workload-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:06:19.910581 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.910551    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"]
Sep 11 22:06:19.918299 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:19.918275    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6f399ea1-openshift-default-5f8c89595b-p9wff"]
Sep 11 22:06:20.984101 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:20.984062    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:06:21.338413 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:21.338313    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f678d2a9-5641-400a-8fac-8b1a026eb82a" path="/var/lib/kubelet/pods/f678d2a9-5641-400a-8fac-8b1a026eb82a/volumes"
Sep 11 22:06:41.480170 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:41.480124    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:06:43.719759 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:43.719676    2587 generic.go:356] "Generic (PLEG): container finished" podID="f8030325-9dfe-4d58-9d37-5baba7ceee45" containerID="4f856e97e9e9822d0693e1b6518c1efe7f1af1ef4d72ab9e3e73672fa8f29d9f" exitCode=0
Sep 11 22:06:43.719759 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:43.719726    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q" event={"ID":"f8030325-9dfe-4d58-9d37-5baba7ceee45","Type":"ContainerDied","Data":"4f856e97e9e9822d0693e1b6518c1efe7f1af1ef4d72ab9e3e73672fa8f29d9f"}
Sep 11 22:06:43.720212 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:43.720128    2587 scope.go:117] "RemoveContainer" containerID="4f856e97e9e9822d0693e1b6518c1efe7f1af1ef4d72ab9e3e73672fa8f29d9f"
Sep 11 22:06:43.721492 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:43.721455    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:06:43.721621 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:43.721545    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:06:43.721621 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:43.721589    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:06:44.731466 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:44.731428    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q" event={"ID":"f8030325-9dfe-4d58-9d37-5baba7ceee45","Type":"ContainerStarted","Data":"5ac889ec6d14993aecd3455a842d6e485594ebdd083f5f1b095b83fac3fc8fa4"}
Sep 11 22:06:45.364184 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:45.364144    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:45.365519 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:45.365492    2587 patch_prober.go:28] interesting pod/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 11 22:06:45.365662 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:45.365549    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q" podUID="f8030325-9dfe-4d58-9d37-5baba7ceee45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 11 22:06:46.363984 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:46.363950    2587 patch_prober.go:28] interesting pod/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 11 22:06:46.364457 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:46.364008    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q" podUID="f8030325-9dfe-4d58-9d37-5baba7ceee45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 11 22:06:47.364560 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:47.364523    2587 patch_prober.go:28] interesting pod/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 11 22:06:47.365060 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:47.364576    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q" podUID="f8030325-9dfe-4d58-9d37-5baba7ceee45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 11 22:06:47.569493 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:47.569451    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:48.364078 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:48.364040    2587 patch_prober.go:28] interesting pod/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 11 22:06:48.364273 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:48.364103    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q" podUID="f8030325-9dfe-4d58-9d37-5baba7ceee45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 11 22:06:49.364528 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:49.364488    2587 patch_prober.go:28] interesting pod/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 11 22:06:49.364935 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:49.364542    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q" podUID="f8030325-9dfe-4d58-9d37-5baba7ceee45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 11 22:06:50.368903 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:50.368866    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:50.370152 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:50.370130    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:06:58.994772 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:58.994734    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp"]
Sep 11 22:06:58.995575 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:58.995533    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp" podUID="c13929af-4c3f-42f2-9179-716de50abacb" containerName="maas-api" containerID="cri-o://0a2f451dd8b13cfb1b98a4ff07ace84d3f27e79e5d8937258023cf59dabb8df0" gracePeriod=30
Sep 11 22:06:59.249620 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:59.249546    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp"
Sep 11 22:06:59.369527 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:59.369493    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q2gr\" (UniqueName: \"kubernetes.io/projected/c13929af-4c3f-42f2-9179-716de50abacb-kube-api-access-9q2gr\") pod \"c13929af-4c3f-42f2-9179-716de50abacb\" (UID: \"c13929af-4c3f-42f2-9179-716de50abacb\") "
Sep 11 22:06:59.369718 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:59.369607    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c13929af-4c3f-42f2-9179-716de50abacb-maas-api-tls\") pod \"c13929af-4c3f-42f2-9179-716de50abacb\" (UID: \"c13929af-4c3f-42f2-9179-716de50abacb\") "
Sep 11 22:06:59.369718 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:59.369628    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13929af-4c3f-42f2-9179-716de50abacb-openshift-service-ca\") pod \"c13929af-4c3f-42f2-9179-716de50abacb\" (UID: \"c13929af-4c3f-42f2-9179-716de50abacb\") "
Sep 11 22:06:59.369718 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:59.369665    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c13929af-4c3f-42f2-9179-716de50abacb-metrics-certs\") pod \"c13929af-4c3f-42f2-9179-716de50abacb\" (UID: \"c13929af-4c3f-42f2-9179-716de50abacb\") "
Sep 11 22:06:59.370031 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:59.370003    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13929af-4c3f-42f2-9179-716de50abacb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c13929af-4c3f-42f2-9179-716de50abacb" (UID: "c13929af-4c3f-42f2-9179-716de50abacb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:06:59.371849 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:59.371818    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c13929af-4c3f-42f2-9179-716de50abacb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c13929af-4c3f-42f2-9179-716de50abacb" (UID: "c13929af-4c3f-42f2-9179-716de50abacb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:06:59.371849 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:59.371835    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c13929af-4c3f-42f2-9179-716de50abacb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c13929af-4c3f-42f2-9179-716de50abacb" (UID: "c13929af-4c3f-42f2-9179-716de50abacb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:06:59.372684 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:59.372655    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13929af-4c3f-42f2-9179-716de50abacb-kube-api-access-9q2gr" (OuterVolumeSpecName: "kube-api-access-9q2gr") pod "c13929af-4c3f-42f2-9179-716de50abacb" (UID: "c13929af-4c3f-42f2-9179-716de50abacb"). InnerVolumeSpecName "kube-api-access-9q2gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:06:59.471641 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:59.471599    2587 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c13929af-4c3f-42f2-9179-716de50abacb-metrics-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:06:59.471641 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:59.471642    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q2gr\" (UniqueName: \"kubernetes.io/projected/c13929af-4c3f-42f2-9179-716de50abacb-kube-api-access-9q2gr\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:06:59.471854 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:59.471659    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c13929af-4c3f-42f2-9179-716de50abacb-maas-api-tls\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:06:59.471854 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:59.471673    2587 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c13929af-4c3f-42f2-9179-716de50abacb-openshift-service-ca\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:06:59.823052 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:59.823011    2587 generic.go:356] "Generic (PLEG): container finished" podID="c13929af-4c3f-42f2-9179-716de50abacb" containerID="0a2f451dd8b13cfb1b98a4ff07ace84d3f27e79e5d8937258023cf59dabb8df0" exitCode=0
Sep 11 22:06:59.823251 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:59.823093    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp"
Sep 11 22:06:59.823251 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:59.823093    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp" event={"ID":"c13929af-4c3f-42f2-9179-716de50abacb","Type":"ContainerDied","Data":"0a2f451dd8b13cfb1b98a4ff07ace84d3f27e79e5d8937258023cf59dabb8df0"}
Sep 11 22:06:59.823251 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:59.823219    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp" event={"ID":"c13929af-4c3f-42f2-9179-716de50abacb","Type":"ContainerDied","Data":"9d9d742e38cf060cfa0d4619f338b78be4d0cea211f941f4165df7759b858902"}
Sep 11 22:06:59.823416 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:59.823253    2587 scope.go:117] "RemoveContainer" containerID="0a2f451dd8b13cfb1b98a4ff07ace84d3f27e79e5d8937258023cf59dabb8df0"
Sep 11 22:06:59.833238 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:59.833219    2587 scope.go:117] "RemoveContainer" containerID="0a2f451dd8b13cfb1b98a4ff07ace84d3f27e79e5d8937258023cf59dabb8df0"
Sep 11 22:06:59.833590 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:06:59.833567    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a2f451dd8b13cfb1b98a4ff07ace84d3f27e79e5d8937258023cf59dabb8df0\": container with ID starting with 0a2f451dd8b13cfb1b98a4ff07ace84d3f27e79e5d8937258023cf59dabb8df0 not found: ID does not exist" containerID="0a2f451dd8b13cfb1b98a4ff07ace84d3f27e79e5d8937258023cf59dabb8df0"
Sep 11 22:06:59.833674 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:59.833604    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a2f451dd8b13cfb1b98a4ff07ace84d3f27e79e5d8937258023cf59dabb8df0"} err="failed to get container status \"0a2f451dd8b13cfb1b98a4ff07ace84d3f27e79e5d8937258023cf59dabb8df0\": rpc error: code = NotFound desc = could not find container \"0a2f451dd8b13cfb1b98a4ff07ace84d3f27e79e5d8937258023cf59dabb8df0\": container with ID starting with 0a2f451dd8b13cfb1b98a4ff07ace84d3f27e79e5d8937258023cf59dabb8df0 not found: ID does not exist"
Sep 11 22:06:59.846525 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:06:59.846492    2587 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13929af_4c3f_42f2_9179_716de50abacb.slice/crio-9d9d742e38cf060cfa0d4619f338b78be4d0cea211f941f4165df7759b858902\": RecentStats: unable to find data in memory cache]"
Sep 11 22:06:59.849782 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:59.849757    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp"]
Sep 11 22:06:59.855036 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:06:59.855012    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-7d5ec8-6548b9c948-b4smp"]
Sep 11 22:07:01.336860 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:01.336820    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c13929af-4c3f-42f2-9179-716de50abacb" path="/var/lib/kubelet/pods/c13929af-4c3f-42f2-9179-716de50abacb/volumes"
Sep 11 22:07:04.217960 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:04.217926    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"]
Sep 11 22:07:04.218479 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:04.218202    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q" podUID="f8030325-9dfe-4d58-9d37-5baba7ceee45" containerName="istio-proxy" containerID="cri-o://5ac889ec6d14993aecd3455a842d6e485594ebdd083f5f1b095b83fac3fc8fa4" gracePeriod=30
Sep 11 22:07:05.199127 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:05.199093    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:07:05.371742 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:05.371709    2587 patch_prober.go:28] interesting pod/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 22:07:05.372102 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:05.371764    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q" podUID="f8030325-9dfe-4d58-9d37-5baba7ceee45" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 22:07:09.485551 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.485491    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:07:09.567739 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.567704    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-istio-data\") pod \"f8030325-9dfe-4d58-9d37-5baba7ceee45\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") "
Sep 11 22:07:09.567931 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.567747    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-workload-socket\") pod \"f8030325-9dfe-4d58-9d37-5baba7ceee45\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") "
Sep 11 22:07:09.567931 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.567796    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8030325-9dfe-4d58-9d37-5baba7ceee45-istio-token\") pod \"f8030325-9dfe-4d58-9d37-5baba7ceee45\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") "
Sep 11 22:07:09.567931 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.567823    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-credential-socket\") pod \"f8030325-9dfe-4d58-9d37-5baba7ceee45\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") "
Sep 11 22:07:09.567931 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.567852    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdzdd\" (UniqueName: \"kubernetes.io/projected/f8030325-9dfe-4d58-9d37-5baba7ceee45-kube-api-access-fdzdd\") pod \"f8030325-9dfe-4d58-9d37-5baba7ceee45\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") "
Sep 11 22:07:09.568153 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.567944    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-workload-certs\") pod \"f8030325-9dfe-4d58-9d37-5baba7ceee45\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") "
Sep 11 22:07:09.568153 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.567972    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8030325-9dfe-4d58-9d37-5baba7ceee45-istio-podinfo\") pod \"f8030325-9dfe-4d58-9d37-5baba7ceee45\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") "
Sep 11 22:07:09.568153 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.568016    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-istio-envoy\") pod \"f8030325-9dfe-4d58-9d37-5baba7ceee45\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") "
Sep 11 22:07:09.568153 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.568045    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-istio-data" (OuterVolumeSpecName: "istio-data") pod "f8030325-9dfe-4d58-9d37-5baba7ceee45" (UID: "f8030325-9dfe-4d58-9d37-5baba7ceee45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:07:09.568153 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.568079    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8030325-9dfe-4d58-9d37-5baba7ceee45-istiod-ca-cert\") pod \"f8030325-9dfe-4d58-9d37-5baba7ceee45\" (UID: \"f8030325-9dfe-4d58-9d37-5baba7ceee45\") "
Sep 11 22:07:09.568153 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.568106    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f8030325-9dfe-4d58-9d37-5baba7ceee45" (UID: "f8030325-9dfe-4d58-9d37-5baba7ceee45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:07:09.568480 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.568205    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f8030325-9dfe-4d58-9d37-5baba7ceee45" (UID: "f8030325-9dfe-4d58-9d37-5baba7ceee45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:07:09.568480 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.568404    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f8030325-9dfe-4d58-9d37-5baba7ceee45" (UID: "f8030325-9dfe-4d58-9d37-5baba7ceee45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:07:09.568597 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.568555    2587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-workload-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:07:09.568597 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.568575    2587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-istio-data\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:07:09.568597 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.568590    2587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-workload-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:07:09.568744 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.568605    2587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-credential-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:07:09.568744 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.568583    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8030325-9dfe-4d58-9d37-5baba7ceee45-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f8030325-9dfe-4d58-9d37-5baba7ceee45" (UID: "f8030325-9dfe-4d58-9d37-5baba7ceee45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:07:09.570344 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.570298    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8030325-9dfe-4d58-9d37-5baba7ceee45-kube-api-access-fdzdd" (OuterVolumeSpecName: "kube-api-access-fdzdd") pod "f8030325-9dfe-4d58-9d37-5baba7ceee45" (UID: "f8030325-9dfe-4d58-9d37-5baba7ceee45"). InnerVolumeSpecName "kube-api-access-fdzdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:07:09.570470 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.570361    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f8030325-9dfe-4d58-9d37-5baba7ceee45" (UID: "f8030325-9dfe-4d58-9d37-5baba7ceee45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:07:09.570470 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.570365    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8030325-9dfe-4d58-9d37-5baba7ceee45-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f8030325-9dfe-4d58-9d37-5baba7ceee45" (UID: "f8030325-9dfe-4d58-9d37-5baba7ceee45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:07:09.570470 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.570433    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8030325-9dfe-4d58-9d37-5baba7ceee45-istio-token" (OuterVolumeSpecName: "istio-token") pod "f8030325-9dfe-4d58-9d37-5baba7ceee45" (UID: "f8030325-9dfe-4d58-9d37-5baba7ceee45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:07:09.669689 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.669655    2587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8030325-9dfe-4d58-9d37-5baba7ceee45-istiod-ca-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:07:09.669689 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.669682    2587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8030325-9dfe-4d58-9d37-5baba7ceee45-istio-token\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:07:09.669689 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.669691    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdzdd\" (UniqueName: \"kubernetes.io/projected/f8030325-9dfe-4d58-9d37-5baba7ceee45-kube-api-access-fdzdd\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:07:09.669960 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.669701    2587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8030325-9dfe-4d58-9d37-5baba7ceee45-istio-podinfo\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:07:09.669960 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.669709    2587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8030325-9dfe-4d58-9d37-5baba7ceee45-istio-envoy\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:07:09.884884 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.882952    2587 generic.go:356] "Generic (PLEG): container finished" podID="f8030325-9dfe-4d58-9d37-5baba7ceee45" containerID="5ac889ec6d14993aecd3455a842d6e485594ebdd083f5f1b095b83fac3fc8fa4" exitCode=0
Sep 11 22:07:09.884884 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.883059    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q" event={"ID":"f8030325-9dfe-4d58-9d37-5baba7ceee45","Type":"ContainerDied","Data":"5ac889ec6d14993aecd3455a842d6e485594ebdd083f5f1b095b83fac3fc8fa4"}
Sep 11 22:07:09.884884 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.883149    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q" event={"ID":"f8030325-9dfe-4d58-9d37-5baba7ceee45","Type":"ContainerDied","Data":"3f1577f7bf07e71234f360400d03514cad83079d8ae3ae0af10a9e1fb989d82b"}
Sep 11 22:07:09.884884 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.883188    2587 scope.go:117] "RemoveContainer" containerID="5ac889ec6d14993aecd3455a842d6e485594ebdd083f5f1b095b83fac3fc8fa4"
Sep 11 22:07:09.884884 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.883412    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"
Sep 11 22:07:09.897794 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.897774    2587 scope.go:117] "RemoveContainer" containerID="4f856e97e9e9822d0693e1b6518c1efe7f1af1ef4d72ab9e3e73672fa8f29d9f"
Sep 11 22:07:09.905947 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.905927    2587 scope.go:117] "RemoveContainer" containerID="5ac889ec6d14993aecd3455a842d6e485594ebdd083f5f1b095b83fac3fc8fa4"
Sep 11 22:07:09.906249 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:07:09.906222    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac889ec6d14993aecd3455a842d6e485594ebdd083f5f1b095b83fac3fc8fa4\": container with ID starting with 5ac889ec6d14993aecd3455a842d6e485594ebdd083f5f1b095b83fac3fc8fa4 not found: ID does not exist" containerID="5ac889ec6d14993aecd3455a842d6e485594ebdd083f5f1b095b83fac3fc8fa4"
Sep 11 22:07:09.906362 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.906253    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac889ec6d14993aecd3455a842d6e485594ebdd083f5f1b095b83fac3fc8fa4"} err="failed to get container status \"5ac889ec6d14993aecd3455a842d6e485594ebdd083f5f1b095b83fac3fc8fa4\": rpc error: code = NotFound desc = could not find container \"5ac889ec6d14993aecd3455a842d6e485594ebdd083f5f1b095b83fac3fc8fa4\": container with ID starting with 5ac889ec6d14993aecd3455a842d6e485594ebdd083f5f1b095b83fac3fc8fa4 not found: ID does not exist"
Sep 11 22:07:09.906362 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.906271    2587 scope.go:117] "RemoveContainer" containerID="4f856e97e9e9822d0693e1b6518c1efe7f1af1ef4d72ab9e3e73672fa8f29d9f"
Sep 11 22:07:09.906568 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:07:09.906544    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f856e97e9e9822d0693e1b6518c1efe7f1af1ef4d72ab9e3e73672fa8f29d9f\": container with ID starting with 4f856e97e9e9822d0693e1b6518c1efe7f1af1ef4d72ab9e3e73672fa8f29d9f not found: ID does not exist" containerID="4f856e97e9e9822d0693e1b6518c1efe7f1af1ef4d72ab9e3e73672fa8f29d9f"
Sep 11 22:07:09.906611 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.906574    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f856e97e9e9822d0693e1b6518c1efe7f1af1ef4d72ab9e3e73672fa8f29d9f"} err="failed to get container status \"4f856e97e9e9822d0693e1b6518c1efe7f1af1ef4d72ab9e3e73672fa8f29d9f\": rpc error: code = NotFound desc = could not find container \"4f856e97e9e9822d0693e1b6518c1efe7f1af1ef4d72ab9e3e73672fa8f29d9f\": container with ID starting with 4f856e97e9e9822d0693e1b6518c1efe7f1af1ef4d72ab9e3e73672fa8f29d9f not found: ID does not exist"
Sep 11 22:07:09.909463 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.909441    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"]
Sep 11 22:07:09.913048 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:09.913028    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-2c9db036-openshift-default-6dbd444c7b-rg48q"]
Sep 11 22:07:11.336557 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:11.336525    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8030325-9dfe-4d58-9d37-5baba7ceee45" path="/var/lib/kubelet/pods/f8030325-9dfe-4d58-9d37-5baba7ceee45/volumes"
Sep 11 22:07:22.169885 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:22.169792    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:07:27.344764 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:27.344738    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw6qf_7ab8a820-c9ec-415c-bfaa-3c9744ac040a/ovn-acl-logging/0.log"
Sep 11 22:07:27.348167 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:27.348145    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw6qf_7ab8a820-c9ec-415c-bfaa-3c9744ac040a/ovn-acl-logging/0.log"
Sep 11 22:07:36.272921 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:36.272882    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:07:42.420036 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.419999    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"]
Sep 11 22:07:42.420793 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.420773    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13929af-4c3f-42f2-9179-716de50abacb" containerName="maas-api"
Sep 11 22:07:42.420997 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.420982    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13929af-4c3f-42f2-9179-716de50abacb" containerName="maas-api"
Sep 11 22:07:42.421162 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.421147    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f678d2a9-5641-400a-8fac-8b1a026eb82a" containerName="istio-proxy"
Sep 11 22:07:42.421259 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.421181    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="f678d2a9-5641-400a-8fac-8b1a026eb82a" containerName="istio-proxy"
Sep 11 22:07:42.421259 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.421199    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8030325-9dfe-4d58-9d37-5baba7ceee45" containerName="istio-proxy"
Sep 11 22:07:42.421259 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.421208    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8030325-9dfe-4d58-9d37-5baba7ceee45" containerName="istio-proxy"
Sep 11 22:07:42.421259 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.421235    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8030325-9dfe-4d58-9d37-5baba7ceee45" containerName="istio-proxy"
Sep 11 22:07:42.421259 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.421243    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8030325-9dfe-4d58-9d37-5baba7ceee45" containerName="istio-proxy"
Sep 11 22:07:42.421655 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.421382    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="c13929af-4c3f-42f2-9179-716de50abacb" containerName="maas-api"
Sep 11 22:07:42.421655 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.421395    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8030325-9dfe-4d58-9d37-5baba7ceee45" containerName="istio-proxy"
Sep 11 22:07:42.421655 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.421409    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="f678d2a9-5641-400a-8fac-8b1a026eb82a" containerName="istio-proxy"
Sep 11 22:07:42.421655 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.421419    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="f678d2a9-5641-400a-8fac-8b1a026eb82a" containerName="istio-proxy"
Sep 11 22:07:42.426444 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.426420    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.429155 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.429136    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-0aea78-openshift-default-dockercfg-cg928\""
Sep 11 22:07:42.437731 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.437702    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"]
Sep 11 22:07:42.502196 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.502158    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-istio-data\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.502408 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.502204    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vrlz\" (UniqueName: \"kubernetes.io/projected/e959efbc-38af-44e0-9077-68de7dec4831-kube-api-access-5vrlz\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.502408 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.502225    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-credential-socket\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.502408 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.502306    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-workload-socket\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.502408 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.502356    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-workload-certs\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.502408 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.502401    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e959efbc-38af-44e0-9077-68de7dec4831-istiod-ca-cert\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.502646 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.502433    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e959efbc-38af-44e0-9077-68de7dec4831-istio-token\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.502646 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.502462    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e959efbc-38af-44e0-9077-68de7dec4831-istio-podinfo\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.502646 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.502579    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-istio-envoy\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.603770 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.603734    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-workload-socket\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.603770 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.603772    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-workload-certs\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.603996 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.603804    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e959efbc-38af-44e0-9077-68de7dec4831-istiod-ca-cert\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.603996 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.603834    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e959efbc-38af-44e0-9077-68de7dec4831-istio-token\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.603996 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.603863    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e959efbc-38af-44e0-9077-68de7dec4831-istio-podinfo\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.603996 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.603948    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-istio-envoy\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.603996 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.603985    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-istio-data\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.604205 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.604166    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-workload-socket\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.604205 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.604173    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vrlz\" (UniqueName: \"kubernetes.io/projected/e959efbc-38af-44e0-9077-68de7dec4831-kube-api-access-5vrlz\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.604317 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.604228    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-workload-certs\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.604317 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.604241    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-credential-socket\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.604520 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.604499    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-istio-data\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.604607 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.604566    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-credential-socket\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.604818 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.604800    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e959efbc-38af-44e0-9077-68de7dec4831-istiod-ca-cert\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.606854 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.606827    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-istio-envoy\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.606977 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.606961    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e959efbc-38af-44e0-9077-68de7dec4831-istio-podinfo\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.611707 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.611685    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e959efbc-38af-44e0-9077-68de7dec4831-istio-token\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.612634 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.612609    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vrlz\" (UniqueName: \"kubernetes.io/projected/e959efbc-38af-44e0-9077-68de7dec4831-kube-api-access-5vrlz\") pod \"e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") " pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:42.739050 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:42.738962    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:43.022770 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:43.022740    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"]
Sep 11 22:07:43.024732 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:07:43.024707    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode959efbc_38af_44e0_9077_68de7dec4831.slice/crio-477ff70992946f5d0cf7b97488a67b248cc0d39d1bfadb86c17ae10df21080c3 WatchSource:0}: Error finding container 477ff70992946f5d0cf7b97488a67b248cc0d39d1bfadb86c17ae10df21080c3: Status 404 returned error can't find the container with id 477ff70992946f5d0cf7b97488a67b248cc0d39d1bfadb86c17ae10df21080c3
Sep 11 22:07:43.027569 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:43.027536    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:07:43.027668 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:43.027622    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:07:43.027735 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:43.027663    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:07:43.058888 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:43.058857    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89" event={"ID":"e959efbc-38af-44e0-9077-68de7dec4831","Type":"ContainerStarted","Data":"477ff70992946f5d0cf7b97488a67b248cc0d39d1bfadb86c17ae10df21080c3"}
Sep 11 22:07:44.065321 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:44.065285    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89" event={"ID":"e959efbc-38af-44e0-9077-68de7dec4831","Type":"ContainerStarted","Data":"bb91cb6554553fc74c6c37eba46a31563660d00a1e816485d618047370b1d8f8"}
Sep 11 22:07:44.088492 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:44.088441    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89" podStartSLOduration=2.088424117 podStartE2EDuration="2.088424117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:07:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:07:44.085180793 +0000 UTC m=+917.260257183" watchObservedRunningTime="2026-09-11 22:07:44.088424117 +0000 UTC m=+917.263500487"
Sep 11 22:07:44.739835 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:44.739799    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:44.745046 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:44.745022    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:45.070730 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:45.070650    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:45.071518 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:45.071496    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:07:53.987420 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:07:53.987371    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:08:04.185816 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:04.185777    2587 generic.go:356] "Generic (PLEG): container finished" podID="e959efbc-38af-44e0-9077-68de7dec4831" containerID="bb91cb6554553fc74c6c37eba46a31563660d00a1e816485d618047370b1d8f8" exitCode=0
Sep 11 22:08:04.186248 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:04.185851    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89" event={"ID":"e959efbc-38af-44e0-9077-68de7dec4831","Type":"ContainerDied","Data":"bb91cb6554553fc74c6c37eba46a31563660d00a1e816485d618047370b1d8f8"}
Sep 11 22:08:04.186301 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:04.186277    2587 scope.go:117] "RemoveContainer" containerID="bb91cb6554553fc74c6c37eba46a31563660d00a1e816485d618047370b1d8f8"
Sep 11 22:08:04.187496 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:04.187471    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:08:04.187575 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:04.187543    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:08:04.187624 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:04.187575    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:08:05.193237 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:05.193196    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89" event={"ID":"e959efbc-38af-44e0-9077-68de7dec4831","Type":"ContainerStarted","Data":"43c40e9ccadd6b7219e491b3491440d2a84466104ad0d04697262b0e961e3ccb"}
Sep 11 22:08:05.740124 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:05.740064    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:08:05.740729 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:05.740700    2587 patch_prober.go:28] interesting pod/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 11 22:08:05.740861 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:05.740780    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89" podUID="e959efbc-38af-44e0-9077-68de7dec4831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 11 22:08:06.739991 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:06.739943    2587 patch_prober.go:28] interesting pod/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 11 22:08:06.740545 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:06.740029    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89" podUID="e959efbc-38af-44e0-9077-68de7dec4831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 11 22:08:07.739884 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:07.739848    2587 patch_prober.go:28] interesting pod/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 11 22:08:07.740175 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:07.740142    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89" podUID="e959efbc-38af-44e0-9077-68de7dec4831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 11 22:08:08.740215 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:08.740162    2587 patch_prober.go:28] interesting pod/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 11 22:08:08.740753 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:08.740240    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89" podUID="e959efbc-38af-44e0-9077-68de7dec4831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 11 22:08:09.740194 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:09.740145    2587 patch_prober.go:28] interesting pod/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 11 22:08:09.740422 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:09.740217    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89" podUID="e959efbc-38af-44e0-9077-68de7dec4831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 11 22:08:10.740114 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:10.740069    2587 patch_prober.go:28] interesting pod/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 11 22:08:10.740300 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:10.740139    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89" podUID="e959efbc-38af-44e0-9077-68de7dec4831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 11 22:08:11.744653 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:11.744619    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:08:11.745138 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:11.744927    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:08:11.745739 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:11.745721    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:08:15.772318 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:15.772283    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:08:32.076816 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:32.076777    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:08:39.357804 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.357767    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"]
Sep 11 22:08:39.358356 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.358320    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f678d2a9-5641-400a-8fac-8b1a026eb82a" containerName="istio-proxy"
Sep 11 22:08:39.358508 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.358487    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="f678d2a9-5641-400a-8fac-8b1a026eb82a" containerName="istio-proxy"
Sep 11 22:08:39.358693 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.358674    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8030325-9dfe-4d58-9d37-5baba7ceee45" containerName="istio-proxy"
Sep 11 22:08:39.362213 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.362193    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.365069 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.365047    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-27f26465-openshift-default-dockercfg-t8lgk\""
Sep 11 22:08:39.374173 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.374150    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"]
Sep 11 22:08:39.480718 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.480680    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-credential-socket\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.480891 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.480725    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-istio-envoy\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.480891 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.480744    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d316781-db1d-4702-b7b0-08546248c0e3-istio-token\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.480891 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.480853    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d316781-db1d-4702-b7b0-08546248c0e3-istiod-ca-cert\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.481032 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.480896    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-istio-data\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.481032 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.480955    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-workload-certs\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.481032 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.480977    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-workload-socket\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.481032 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.480994    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq2mw\" (UniqueName: \"kubernetes.io/projected/7d316781-db1d-4702-b7b0-08546248c0e3-kube-api-access-lq2mw\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.481179 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.481037    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d316781-db1d-4702-b7b0-08546248c0e3-istio-podinfo\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.581949 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.581916    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d316781-db1d-4702-b7b0-08546248c0e3-istio-podinfo\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.582131 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.582007    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-credential-socket\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.582131 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.582029    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-istio-envoy\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.582131 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.582043    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d316781-db1d-4702-b7b0-08546248c0e3-istio-token\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.582131 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.582065    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d316781-db1d-4702-b7b0-08546248c0e3-istiod-ca-cert\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.582131 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.582101    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-istio-data\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.582470 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.582133    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-workload-certs\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.582470 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.582162    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-workload-socket\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.582470 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.582185    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq2mw\" (UniqueName: \"kubernetes.io/projected/7d316781-db1d-4702-b7b0-08546248c0e3-kube-api-access-lq2mw\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.582646 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.582606    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-credential-socket\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.582708 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.582644    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-workload-certs\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.582708 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.582681    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-workload-socket\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.582821 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.582729    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-istio-data\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.582930 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.582909    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d316781-db1d-4702-b7b0-08546248c0e3-istiod-ca-cert\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.584504 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.584482    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-istio-envoy\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.584605 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.584556    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d316781-db1d-4702-b7b0-08546248c0e3-istio-podinfo\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.590451 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.590421    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d316781-db1d-4702-b7b0-08546248c0e3-istio-token\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.590634 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.590619    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq2mw\" (UniqueName: \"kubernetes.io/projected/7d316781-db1d-4702-b7b0-08546248c0e3-kube-api-access-lq2mw\") pod \"e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") " pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.674068 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.674031    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:39.954527 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.954503    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"]
Sep 11 22:08:39.956066 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:08:39.956043    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d316781_db1d_4702_b7b0_08546248c0e3.slice/crio-9938688b4147a005c0331612a852c5a2432393c8b088647399f1f1874e06ceca WatchSource:0}: Error finding container 9938688b4147a005c0331612a852c5a2432393c8b088647399f1f1874e06ceca: Status 404 returned error can't find the container with id 9938688b4147a005c0331612a852c5a2432393c8b088647399f1f1874e06ceca
Sep 11 22:08:39.958803 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.958770    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:08:39.958893 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.958833    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:08:39.958893 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:39.958863    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:08:40.393463 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:40.393369    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8" event={"ID":"7d316781-db1d-4702-b7b0-08546248c0e3","Type":"ContainerStarted","Data":"d89a7a19972c340beca60df211c929d2ebf1b49d8bfd0fcaee4aed445719ff80"}
Sep 11 22:08:40.393463 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:40.393423    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8" event={"ID":"7d316781-db1d-4702-b7b0-08546248c0e3","Type":"ContainerStarted","Data":"9938688b4147a005c0331612a852c5a2432393c8b088647399f1f1874e06ceca"}
Sep 11 22:08:40.416294 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:40.416230    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8" podStartSLOduration=1.41621127 podStartE2EDuration="1.41621127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:08:40.413844165 +0000 UTC m=+973.588920534" watchObservedRunningTime="2026-09-11 22:08:40.41621127 +0000 UTC m=+973.591287640"
Sep 11 22:08:40.674499 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:40.674468    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:41.679695 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:41.679668    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:42.404668 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:42.404640    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:42.405505 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:42.405488    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:08:48.426855 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:48.426812    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw"]
Sep 11 22:08:48.431126 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:48.431102    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw"
Sep 11 22:08:48.433839 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:48.433820    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-27f26465-serving-cert\""
Sep 11 22:08:48.444935 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:48.444910    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw"]
Sep 11 22:08:48.580289 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:48.580224    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cebfc583-703d-4b76-81ff-3b876a3ffc17-openshift-service-ca\") pod \"maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw\" (UID: \"cebfc583-703d-4b76-81ff-3b876a3ffc17\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw"
Sep 11 22:08:48.580289 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:48.580290    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cebfc583-703d-4b76-81ff-3b876a3ffc17-metrics-certs\") pod \"maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw\" (UID: \"cebfc583-703d-4b76-81ff-3b876a3ffc17\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw"
Sep 11 22:08:48.580545 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:48.580377    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px5wl\" (UniqueName: \"kubernetes.io/projected/cebfc583-703d-4b76-81ff-3b876a3ffc17-kube-api-access-px5wl\") pod \"maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw\" (UID: \"cebfc583-703d-4b76-81ff-3b876a3ffc17\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw"
Sep 11 22:08:48.580595 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:48.580582    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cebfc583-703d-4b76-81ff-3b876a3ffc17-maas-api-tls\") pod \"maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw\" (UID: \"cebfc583-703d-4b76-81ff-3b876a3ffc17\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw"
Sep 11 22:08:48.681432 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:48.681321    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cebfc583-703d-4b76-81ff-3b876a3ffc17-maas-api-tls\") pod \"maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw\" (UID: \"cebfc583-703d-4b76-81ff-3b876a3ffc17\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw"
Sep 11 22:08:48.681432 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:48.681417    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cebfc583-703d-4b76-81ff-3b876a3ffc17-openshift-service-ca\") pod \"maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw\" (UID: \"cebfc583-703d-4b76-81ff-3b876a3ffc17\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw"
Sep 11 22:08:48.681662 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:48.681442    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cebfc583-703d-4b76-81ff-3b876a3ffc17-metrics-certs\") pod \"maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw\" (UID: \"cebfc583-703d-4b76-81ff-3b876a3ffc17\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw"
Sep 11 22:08:48.681662 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:08:48.681447    2587 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-serving-cert: secret "maas-api-e2e-mt-27f26465-serving-cert" not found
Sep 11 22:08:48.681662 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:48.681465    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px5wl\" (UniqueName: \"kubernetes.io/projected/cebfc583-703d-4b76-81ff-3b876a3ffc17-kube-api-access-px5wl\") pod \"maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw\" (UID: \"cebfc583-703d-4b76-81ff-3b876a3ffc17\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw"
Sep 11 22:08:48.681662 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:08:48.681526    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cebfc583-703d-4b76-81ff-3b876a3ffc17-maas-api-tls podName:cebfc583-703d-4b76-81ff-3b876a3ffc17 nodeName:}" failed. No retries permitted until 2026-09-11 22:08:49.181502771 +0000 UTC m=+982.356579135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cebfc583-703d-4b76-81ff-3b876a3ffc17-maas-api-tls") pod "maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw" (UID: "cebfc583-703d-4b76-81ff-3b876a3ffc17") : secret "maas-api-e2e-mt-27f26465-serving-cert" not found
Sep 11 22:08:48.682076 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:48.682053    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cebfc583-703d-4b76-81ff-3b876a3ffc17-openshift-service-ca\") pod \"maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw\" (UID: \"cebfc583-703d-4b76-81ff-3b876a3ffc17\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw"
Sep 11 22:08:48.684056 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:48.684028    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cebfc583-703d-4b76-81ff-3b876a3ffc17-metrics-certs\") pod \"maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw\" (UID: \"cebfc583-703d-4b76-81ff-3b876a3ffc17\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw"
Sep 11 22:08:48.692913 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:48.692891    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px5wl\" (UniqueName: \"kubernetes.io/projected/cebfc583-703d-4b76-81ff-3b876a3ffc17-kube-api-access-px5wl\") pod \"maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw\" (UID: \"cebfc583-703d-4b76-81ff-3b876a3ffc17\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw"
Sep 11 22:08:49.187469 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:49.187427    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cebfc583-703d-4b76-81ff-3b876a3ffc17-maas-api-tls\") pod \"maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw\" (UID: \"cebfc583-703d-4b76-81ff-3b876a3ffc17\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw"
Sep 11 22:08:49.190873 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:49.190842    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cebfc583-703d-4b76-81ff-3b876a3ffc17-maas-api-tls\") pod \"maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw\" (UID: \"cebfc583-703d-4b76-81ff-3b876a3ffc17\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw"
Sep 11 22:08:49.349638 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:49.349599    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw"
Sep 11 22:08:49.625166 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:49.625123    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw"]
Sep 11 22:08:49.627317 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:08:49.627291    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcebfc583_703d_4b76_81ff_3b876a3ffc17.slice/crio-279ce17771476ea5190403211c542c27a60000fca6e9888a3844dd5ab599699d WatchSource:0}: Error finding container 279ce17771476ea5190403211c542c27a60000fca6e9888a3844dd5ab599699d: Status 404 returned error can't find the container with id 279ce17771476ea5190403211c542c27a60000fca6e9888a3844dd5ab599699d
Sep 11 22:08:50.459249 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:50.459208    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw" event={"ID":"cebfc583-703d-4b76-81ff-3b876a3ffc17","Type":"ContainerStarted","Data":"e88dcc0190af17edca1ee0ba3fe0a7c9a8da832b79eab256a0e0d3829d4eedd8"}
Sep 11 22:08:50.459475 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:50.459283    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw"
Sep 11 22:08:50.459475 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:50.459310    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw" event={"ID":"cebfc583-703d-4b76-81ff-3b876a3ffc17","Type":"ContainerStarted","Data":"279ce17771476ea5190403211c542c27a60000fca6e9888a3844dd5ab599699d"}
Sep 11 22:08:50.479398 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:50.479320    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw" podStartSLOduration=2.117800361 podStartE2EDuration="2.479302891s" totalImagesPullingTime="361.50253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:08:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:08:50.4761489 +0000 UTC m=+983.651225284" watchObservedRunningTime="2026-09-11 22:08:50.479302891 +0000 UTC m=+983.654379272"
Sep 11 22:08:52.575678 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:52.575642    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:08:56.470424 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:08:56.470385    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw"
Sep 11 22:09:18.610131 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:18.610099    2587 generic.go:356] "Generic (PLEG): container finished" podID="7d316781-db1d-4702-b7b0-08546248c0e3" containerID="d89a7a19972c340beca60df211c929d2ebf1b49d8bfd0fcaee4aed445719ff80" exitCode=0
Sep 11 22:09:18.610694 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:18.610150    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8" event={"ID":"7d316781-db1d-4702-b7b0-08546248c0e3","Type":"ContainerDied","Data":"d89a7a19972c340beca60df211c929d2ebf1b49d8bfd0fcaee4aed445719ff80"}
Sep 11 22:09:18.610694 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:18.610662    2587 scope.go:117] "RemoveContainer" containerID="d89a7a19972c340beca60df211c929d2ebf1b49d8bfd0fcaee4aed445719ff80"
Sep 11 22:09:18.611845 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:18.611814    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:09:18.611927 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:18.611883    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:09:18.611927 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:18.611913    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:09:19.618342 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:19.618301    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8" event={"ID":"7d316781-db1d-4702-b7b0-08546248c0e3","Type":"ContainerStarted","Data":"25caebd2df4e85079a7bc61d06375c0a3060cc4e08bc0f3e9913390ab3c84c79"}
Sep 11 22:09:19.674813 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:19.674776    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:09:19.674988 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:19.674933    2587 patch_prober.go:28] interesting pod/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 11 22:09:19.675035 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:19.674983    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8" podUID="7d316781-db1d-4702-b7b0-08546248c0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 11 22:09:20.674721 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:20.674682    2587 patch_prober.go:28] interesting pod/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 11 22:09:20.675124 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:20.674742    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8" podUID="7d316781-db1d-4702-b7b0-08546248c0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 11 22:09:21.674502 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:21.674464    2587 patch_prober.go:28] interesting pod/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 11 22:09:21.674682 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:21.674526    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8" podUID="7d316781-db1d-4702-b7b0-08546248c0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 11 22:09:22.674671 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:22.674632    2587 patch_prober.go:28] interesting pod/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 11 22:09:22.675044 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:22.674709    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8" podUID="7d316781-db1d-4702-b7b0-08546248c0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 11 22:09:23.675448 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:23.675410    2587 patch_prober.go:28] interesting pod/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 11 22:09:23.675863 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:23.675478    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8" podUID="7d316781-db1d-4702-b7b0-08546248c0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 11 22:09:24.675068 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:24.675032    2587 patch_prober.go:28] interesting pod/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 11 22:09:24.675246 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:24.675105    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8" podUID="7d316781-db1d-4702-b7b0-08546248c0e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 11 22:09:25.678783 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:25.678748    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:09:25.679232 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:25.679048    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:09:25.679909 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:25.679887    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:09:26.175612 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:26.175575    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:09:32.971496 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:32.971454    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw"]
Sep 11 22:09:32.972040 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:32.971722    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw" podUID="cebfc583-703d-4b76-81ff-3b876a3ffc17" containerName="maas-api" containerID="cri-o://e88dcc0190af17edca1ee0ba3fe0a7c9a8da832b79eab256a0e0d3829d4eedd8" gracePeriod=30
Sep 11 22:09:33.235668 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:33.235608    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw"
Sep 11 22:09:33.380127 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:33.380089    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cebfc583-703d-4b76-81ff-3b876a3ffc17-maas-api-tls\") pod \"cebfc583-703d-4b76-81ff-3b876a3ffc17\" (UID: \"cebfc583-703d-4b76-81ff-3b876a3ffc17\") "
Sep 11 22:09:33.380350 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:33.380171    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cebfc583-703d-4b76-81ff-3b876a3ffc17-metrics-certs\") pod \"cebfc583-703d-4b76-81ff-3b876a3ffc17\" (UID: \"cebfc583-703d-4b76-81ff-3b876a3ffc17\") "
Sep 11 22:09:33.380350 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:33.380259    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-px5wl\" (UniqueName: \"kubernetes.io/projected/cebfc583-703d-4b76-81ff-3b876a3ffc17-kube-api-access-px5wl\") pod \"cebfc583-703d-4b76-81ff-3b876a3ffc17\" (UID: \"cebfc583-703d-4b76-81ff-3b876a3ffc17\") "
Sep 11 22:09:33.380503 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:33.380380    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cebfc583-703d-4b76-81ff-3b876a3ffc17-openshift-service-ca\") pod \"cebfc583-703d-4b76-81ff-3b876a3ffc17\" (UID: \"cebfc583-703d-4b76-81ff-3b876a3ffc17\") "
Sep 11 22:09:33.381093 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:33.381063    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cebfc583-703d-4b76-81ff-3b876a3ffc17-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cebfc583-703d-4b76-81ff-3b876a3ffc17" (UID: "cebfc583-703d-4b76-81ff-3b876a3ffc17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:09:33.383284 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:33.383245    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cebfc583-703d-4b76-81ff-3b876a3ffc17-kube-api-access-px5wl" (OuterVolumeSpecName: "kube-api-access-px5wl") pod "cebfc583-703d-4b76-81ff-3b876a3ffc17" (UID: "cebfc583-703d-4b76-81ff-3b876a3ffc17"). InnerVolumeSpecName "kube-api-access-px5wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:09:33.385021 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:33.384992    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cebfc583-703d-4b76-81ff-3b876a3ffc17-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cebfc583-703d-4b76-81ff-3b876a3ffc17" (UID: "cebfc583-703d-4b76-81ff-3b876a3ffc17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:09:33.385135 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:33.385069    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cebfc583-703d-4b76-81ff-3b876a3ffc17-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cebfc583-703d-4b76-81ff-3b876a3ffc17" (UID: "cebfc583-703d-4b76-81ff-3b876a3ffc17"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:09:33.481846 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:33.481800    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px5wl\" (UniqueName: \"kubernetes.io/projected/cebfc583-703d-4b76-81ff-3b876a3ffc17-kube-api-access-px5wl\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:09:33.481846 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:33.481838    2587 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cebfc583-703d-4b76-81ff-3b876a3ffc17-openshift-service-ca\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:09:33.481846 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:33.481851    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cebfc583-703d-4b76-81ff-3b876a3ffc17-maas-api-tls\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:09:33.482088 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:33.481862    2587 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cebfc583-703d-4b76-81ff-3b876a3ffc17-metrics-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:09:33.712645 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:33.712612    2587 generic.go:356] "Generic (PLEG): container finished" podID="cebfc583-703d-4b76-81ff-3b876a3ffc17" containerID="e88dcc0190af17edca1ee0ba3fe0a7c9a8da832b79eab256a0e0d3829d4eedd8" exitCode=0
Sep 11 22:09:33.712831 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:33.712654    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw"
Sep 11 22:09:33.712831 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:33.712663    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw" event={"ID":"cebfc583-703d-4b76-81ff-3b876a3ffc17","Type":"ContainerDied","Data":"e88dcc0190af17edca1ee0ba3fe0a7c9a8da832b79eab256a0e0d3829d4eedd8"}
Sep 11 22:09:33.712831 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:33.712714    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw" event={"ID":"cebfc583-703d-4b76-81ff-3b876a3ffc17","Type":"ContainerDied","Data":"279ce17771476ea5190403211c542c27a60000fca6e9888a3844dd5ab599699d"}
Sep 11 22:09:33.712831 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:33.712738    2587 scope.go:117] "RemoveContainer" containerID="e88dcc0190af17edca1ee0ba3fe0a7c9a8da832b79eab256a0e0d3829d4eedd8"
Sep 11 22:09:33.722282 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:33.722259    2587 scope.go:117] "RemoveContainer" containerID="e88dcc0190af17edca1ee0ba3fe0a7c9a8da832b79eab256a0e0d3829d4eedd8"
Sep 11 22:09:33.722610 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:09:33.722586    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e88dcc0190af17edca1ee0ba3fe0a7c9a8da832b79eab256a0e0d3829d4eedd8\": container with ID starting with e88dcc0190af17edca1ee0ba3fe0a7c9a8da832b79eab256a0e0d3829d4eedd8 not found: ID does not exist" containerID="e88dcc0190af17edca1ee0ba3fe0a7c9a8da832b79eab256a0e0d3829d4eedd8"
Sep 11 22:09:33.722663 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:33.722621    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88dcc0190af17edca1ee0ba3fe0a7c9a8da832b79eab256a0e0d3829d4eedd8"} err="failed to get container status \"e88dcc0190af17edca1ee0ba3fe0a7c9a8da832b79eab256a0e0d3829d4eedd8\": rpc error: code = NotFound desc = could not find container \"e88dcc0190af17edca1ee0ba3fe0a7c9a8da832b79eab256a0e0d3829d4eedd8\": container with ID starting with e88dcc0190af17edca1ee0ba3fe0a7c9a8da832b79eab256a0e0d3829d4eedd8 not found: ID does not exist"
Sep 11 22:09:33.736467 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:33.736443    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw"]
Sep 11 22:09:33.740549 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:33.740528    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-27f26465-74d7966f4d-dv6cw"]
Sep 11 22:09:35.340485 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:35.340450    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cebfc583-703d-4b76-81ff-3b876a3ffc17" path="/var/lib/kubelet/pods/cebfc583-703d-4b76-81ff-3b876a3ffc17/volumes"
Sep 11 22:09:39.748491 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:39.748460    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"]
Sep 11 22:09:39.748954 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:39.748676    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8" podUID="7d316781-db1d-4702-b7b0-08546248c0e3" containerName="istio-proxy" containerID="cri-o://25caebd2df4e85079a7bc61d06375c0a3060cc4e08bc0f3e9913390ab3c84c79" gracePeriod=30
Sep 11 22:09:40.682114 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:40.682076    2587 patch_prober.go:28] interesting pod/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 22:09:40.682275 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:40.682132    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8" podUID="7d316781-db1d-4702-b7b0-08546248c0e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 22:09:44.552895 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:44.552856    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"]
Sep 11 22:09:44.553319 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:44.553131    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89" podUID="e959efbc-38af-44e0-9077-68de7dec4831" containerName="istio-proxy" containerID="cri-o://43c40e9ccadd6b7219e491b3491440d2a84466104ad0d04697262b0e961e3ccb" gracePeriod=30
Sep 11 22:09:45.001051 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.001027    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:09:45.108064 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.108023    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-credential-socket\") pod \"7d316781-db1d-4702-b7b0-08546248c0e3\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") "
Sep 11 22:09:45.108064 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.108070    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lq2mw\" (UniqueName: \"kubernetes.io/projected/7d316781-db1d-4702-b7b0-08546248c0e3-kube-api-access-lq2mw\") pod \"7d316781-db1d-4702-b7b0-08546248c0e3\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") "
Sep 11 22:09:45.108314 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.108110    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-workload-certs\") pod \"7d316781-db1d-4702-b7b0-08546248c0e3\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") "
Sep 11 22:09:45.108314 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.108157    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-workload-socket\") pod \"7d316781-db1d-4702-b7b0-08546248c0e3\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") "
Sep 11 22:09:45.108314 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.108190    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d316781-db1d-4702-b7b0-08546248c0e3-istiod-ca-cert\") pod \"7d316781-db1d-4702-b7b0-08546248c0e3\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") "
Sep 11 22:09:45.108521 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.108375    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7d316781-db1d-4702-b7b0-08546248c0e3" (UID: "7d316781-db1d-4702-b7b0-08546248c0e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:09:45.108521 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.108394    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d316781-db1d-4702-b7b0-08546248c0e3-istio-token\") pod \"7d316781-db1d-4702-b7b0-08546248c0e3\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") "
Sep 11 22:09:45.108521 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.108476    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-istio-envoy\") pod \"7d316781-db1d-4702-b7b0-08546248c0e3\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") "
Sep 11 22:09:45.108521 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.108473    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7d316781-db1d-4702-b7b0-08546248c0e3" (UID: "7d316781-db1d-4702-b7b0-08546248c0e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:09:45.108521 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.108517    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-istio-data\") pod \"7d316781-db1d-4702-b7b0-08546248c0e3\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") "
Sep 11 22:09:45.108779 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.108485    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7d316781-db1d-4702-b7b0-08546248c0e3" (UID: "7d316781-db1d-4702-b7b0-08546248c0e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:09:45.108779 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.108544    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d316781-db1d-4702-b7b0-08546248c0e3-istio-podinfo\") pod \"7d316781-db1d-4702-b7b0-08546248c0e3\" (UID: \"7d316781-db1d-4702-b7b0-08546248c0e3\") "
Sep 11 22:09:45.108779 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.108627    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d316781-db1d-4702-b7b0-08546248c0e3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7d316781-db1d-4702-b7b0-08546248c0e3" (UID: "7d316781-db1d-4702-b7b0-08546248c0e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:09:45.108779 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.108720    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-istio-data" (OuterVolumeSpecName: "istio-data") pod "7d316781-db1d-4702-b7b0-08546248c0e3" (UID: "7d316781-db1d-4702-b7b0-08546248c0e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:09:45.108935 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.108890    2587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-istio-data\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:09:45.108935 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.108917    2587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-credential-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:09:45.109005 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.108935    2587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-workload-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:09:45.109005 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.108949    2587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-workload-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:09:45.109005 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.108961    2587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d316781-db1d-4702-b7b0-08546248c0e3-istiod-ca-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:09:45.110442 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.110417    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d316781-db1d-4702-b7b0-08546248c0e3-kube-api-access-lq2mw" (OuterVolumeSpecName: "kube-api-access-lq2mw") pod "7d316781-db1d-4702-b7b0-08546248c0e3" (UID: "7d316781-db1d-4702-b7b0-08546248c0e3"). InnerVolumeSpecName "kube-api-access-lq2mw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:09:45.110592 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.110569    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d316781-db1d-4702-b7b0-08546248c0e3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7d316781-db1d-4702-b7b0-08546248c0e3" (UID: "7d316781-db1d-4702-b7b0-08546248c0e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:09:45.110851 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.110828    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d316781-db1d-4702-b7b0-08546248c0e3-istio-token" (OuterVolumeSpecName: "istio-token") pod "7d316781-db1d-4702-b7b0-08546248c0e3" (UID: "7d316781-db1d-4702-b7b0-08546248c0e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:09:45.110851 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.110843    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7d316781-db1d-4702-b7b0-08546248c0e3" (UID: "7d316781-db1d-4702-b7b0-08546248c0e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:09:45.209758 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.209720    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq2mw\" (UniqueName: \"kubernetes.io/projected/7d316781-db1d-4702-b7b0-08546248c0e3-kube-api-access-lq2mw\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:09:45.209758 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.209752    2587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d316781-db1d-4702-b7b0-08546248c0e3-istio-token\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:09:45.209758 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.209763    2587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d316781-db1d-4702-b7b0-08546248c0e3-istio-envoy\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:09:45.209994 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.209771    2587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d316781-db1d-4702-b7b0-08546248c0e3-istio-podinfo\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:09:45.291590 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.291553    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"]
Sep 11 22:09:45.292123 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.292104    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d316781-db1d-4702-b7b0-08546248c0e3" containerName="istio-proxy"
Sep 11 22:09:45.292123 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.292123    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d316781-db1d-4702-b7b0-08546248c0e3" containerName="istio-proxy"
Sep 11 22:09:45.292259 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.292138    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d316781-db1d-4702-b7b0-08546248c0e3" containerName="istio-proxy"
Sep 11 22:09:45.292259 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.292145    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d316781-db1d-4702-b7b0-08546248c0e3" containerName="istio-proxy"
Sep 11 22:09:45.292259 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.292156    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cebfc583-703d-4b76-81ff-3b876a3ffc17" containerName="maas-api"
Sep 11 22:09:45.292259 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.292162    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="cebfc583-703d-4b76-81ff-3b876a3ffc17" containerName="maas-api"
Sep 11 22:09:45.292259 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.292223    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d316781-db1d-4702-b7b0-08546248c0e3" containerName="istio-proxy"
Sep 11 22:09:45.292259 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.292230    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d316781-db1d-4702-b7b0-08546248c0e3" containerName="istio-proxy"
Sep 11 22:09:45.292259 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.292240    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="cebfc583-703d-4b76-81ff-3b876a3ffc17" containerName="maas-api"
Sep 11 22:09:45.295881 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.295846    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.298792 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.298769    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-26d968-openshift-default-dockercfg-vq496\""
Sep 11 22:09:45.308183 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.308140    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"]
Sep 11 22:09:45.411405 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.411270    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-workload-certs\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.411405 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.411323    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istio-podinfo\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.411618 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.411452    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istio-data\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.411618 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.411481    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj5rs\" (UniqueName: \"kubernetes.io/projected/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-kube-api-access-pj5rs\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.411618 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.411524    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-workload-socket\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.411618 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.411554    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istio-token\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.411618 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.411591    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istiod-ca-cert\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.411618 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.411615    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-credential-socket\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.411834 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.411739    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istio-envoy\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.513142 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.513103    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-workload-certs\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.513142 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.513147    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istio-podinfo\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.513421 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.513190    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istio-data\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.513421 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.513208    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj5rs\" (UniqueName: \"kubernetes.io/projected/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-kube-api-access-pj5rs\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.513421 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.513237    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-workload-socket\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.513421 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.513252    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istio-token\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.513421 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.513385    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istiod-ca-cert\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.513692 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.513437    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-credential-socket\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.513692 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.513547    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istio-envoy\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.513692 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.513612    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-workload-certs\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.513921 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.513702    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-workload-socket\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.513921 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.513746    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istio-data\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.514037 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.514017    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-credential-socket\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.514176 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.514159    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istiod-ca-cert\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.515684 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.515665    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istio-envoy\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.515771 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.515751    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istio-podinfo\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.521882 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.521858    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istio-token\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.522077 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.522055    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj5rs\" (UniqueName: \"kubernetes.io/projected/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-kube-api-access-pj5rs\") pod \"e2e-noten-26d968-openshift-default-845977b986-v5hcg\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") " pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.606614 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.606578    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:45.783371 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.783339    2587 generic.go:356] "Generic (PLEG): container finished" podID="7d316781-db1d-4702-b7b0-08546248c0e3" containerID="25caebd2df4e85079a7bc61d06375c0a3060cc4e08bc0f3e9913390ab3c84c79" exitCode=0
Sep 11 22:09:45.783609 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.783390    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"
Sep 11 22:09:45.783609 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.783424    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8" event={"ID":"7d316781-db1d-4702-b7b0-08546248c0e3","Type":"ContainerDied","Data":"25caebd2df4e85079a7bc61d06375c0a3060cc4e08bc0f3e9913390ab3c84c79"}
Sep 11 22:09:45.783609 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.783496    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8" event={"ID":"7d316781-db1d-4702-b7b0-08546248c0e3","Type":"ContainerDied","Data":"9938688b4147a005c0331612a852c5a2432393c8b088647399f1f1874e06ceca"}
Sep 11 22:09:45.783609 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.783544    2587 scope.go:117] "RemoveContainer" containerID="25caebd2df4e85079a7bc61d06375c0a3060cc4e08bc0f3e9913390ab3c84c79"
Sep 11 22:09:45.793574 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.793553    2587 scope.go:117] "RemoveContainer" containerID="d89a7a19972c340beca60df211c929d2ebf1b49d8bfd0fcaee4aed445719ff80"
Sep 11 22:09:45.801948 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.801920    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"]
Sep 11 22:09:45.802247 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.802228    2587 scope.go:117] "RemoveContainer" containerID="25caebd2df4e85079a7bc61d06375c0a3060cc4e08bc0f3e9913390ab3c84c79"
Sep 11 22:09:45.803009 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:09:45.802939    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25caebd2df4e85079a7bc61d06375c0a3060cc4e08bc0f3e9913390ab3c84c79\": container with ID starting with 25caebd2df4e85079a7bc61d06375c0a3060cc4e08bc0f3e9913390ab3c84c79 not found: ID does not exist" containerID="25caebd2df4e85079a7bc61d06375c0a3060cc4e08bc0f3e9913390ab3c84c79"
Sep 11 22:09:45.803009 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.802976    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25caebd2df4e85079a7bc61d06375c0a3060cc4e08bc0f3e9913390ab3c84c79"} err="failed to get container status \"25caebd2df4e85079a7bc61d06375c0a3060cc4e08bc0f3e9913390ab3c84c79\": rpc error: code = NotFound desc = could not find container \"25caebd2df4e85079a7bc61d06375c0a3060cc4e08bc0f3e9913390ab3c84c79\": container with ID starting with 25caebd2df4e85079a7bc61d06375c0a3060cc4e08bc0f3e9913390ab3c84c79 not found: ID does not exist"
Sep 11 22:09:45.803009 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.802998    2587 scope.go:117] "RemoveContainer" containerID="d89a7a19972c340beca60df211c929d2ebf1b49d8bfd0fcaee4aed445719ff80"
Sep 11 22:09:45.803304 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:09:45.803286    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d89a7a19972c340beca60df211c929d2ebf1b49d8bfd0fcaee4aed445719ff80\": container with ID starting with d89a7a19972c340beca60df211c929d2ebf1b49d8bfd0fcaee4aed445719ff80 not found: ID does not exist" containerID="d89a7a19972c340beca60df211c929d2ebf1b49d8bfd0fcaee4aed445719ff80"
Sep 11 22:09:45.803376 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.803312    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89a7a19972c340beca60df211c929d2ebf1b49d8bfd0fcaee4aed445719ff80"} err="failed to get container status \"d89a7a19972c340beca60df211c929d2ebf1b49d8bfd0fcaee4aed445719ff80\": rpc error: code = NotFound desc = could not find container \"d89a7a19972c340beca60df211c929d2ebf1b49d8bfd0fcaee4aed445719ff80\": container with ID starting with d89a7a19972c340beca60df211c929d2ebf1b49d8bfd0fcaee4aed445719ff80 not found: ID does not exist"
Sep 11 22:09:45.806536 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.806516    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-27f26465-openshift-default-5c78f6bff6-sqbm8"]
Sep 11 22:09:45.884540 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.884514    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"]
Sep 11 22:09:45.886163 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:09:45.886129    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb68ba8c5_24f8_4a61_ae26_e025fa65f52f.slice/crio-bc8eef87148282819b685f614746d4462a8a26bace31e64635d5ebdf7024c230 WatchSource:0}: Error finding container bc8eef87148282819b685f614746d4462a8a26bace31e64635d5ebdf7024c230: Status 404 returned error can't find the container with id bc8eef87148282819b685f614746d4462a8a26bace31e64635d5ebdf7024c230
Sep 11 22:09:45.888769 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.888730    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:09:45.888873 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.888815    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:09:45.888873 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:45.888854    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:09:46.791807 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:46.791767    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg" event={"ID":"b68ba8c5-24f8-4a61-ae26-e025fa65f52f","Type":"ContainerStarted","Data":"5baffda5eb1dd95379275440961e456f754ed4b7aa8b8dc41c1103539448390f"}
Sep 11 22:09:46.792211 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:46.791826    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg" event={"ID":"b68ba8c5-24f8-4a61-ae26-e025fa65f52f","Type":"ContainerStarted","Data":"bc8eef87148282819b685f614746d4462a8a26bace31e64635d5ebdf7024c230"}
Sep 11 22:09:46.815846 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:46.815787    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg" podStartSLOduration=1.8157719669999999 podStartE2EDuration="1.815771967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:09:46.812268148 +0000 UTC m=+1039.987344536" watchObservedRunningTime="2026-09-11 22:09:46.815771967 +0000 UTC m=+1039.990848336"
Sep 11 22:09:47.342263 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:47.342219    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d316781-db1d-4702-b7b0-08546248c0e3" path="/var/lib/kubelet/pods/7d316781-db1d-4702-b7b0-08546248c0e3/volumes"
Sep 11 22:09:47.607603 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:47.607512    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:47.612546 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:47.612518    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:47.797269 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:47.797231    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:47.798320 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:47.798298    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:09:49.808555 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.808531    2587 generic.go:356] "Generic (PLEG): container finished" podID="e959efbc-38af-44e0-9077-68de7dec4831" containerID="43c40e9ccadd6b7219e491b3491440d2a84466104ad0d04697262b0e961e3ccb" exitCode=0
Sep 11 22:09:49.808851 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.808600    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89" event={"ID":"e959efbc-38af-44e0-9077-68de7dec4831","Type":"ContainerDied","Data":"43c40e9ccadd6b7219e491b3491440d2a84466104ad0d04697262b0e961e3ccb"}
Sep 11 22:09:49.808851 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.808661    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89" event={"ID":"e959efbc-38af-44e0-9077-68de7dec4831","Type":"ContainerDied","Data":"477ff70992946f5d0cf7b97488a67b248cc0d39d1bfadb86c17ae10df21080c3"}
Sep 11 22:09:49.808851 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.808688    2587 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="477ff70992946f5d0cf7b97488a67b248cc0d39d1bfadb86c17ae10df21080c3"
Sep 11 22:09:49.808851 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.808703    2587 scope.go:117] "RemoveContainer" containerID="bb91cb6554553fc74c6c37eba46a31563660d00a1e816485d618047370b1d8f8"
Sep 11 22:09:49.827936 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.827910    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:09:49.856078 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.856051    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e959efbc-38af-44e0-9077-68de7dec4831-istio-podinfo\") pod \"e959efbc-38af-44e0-9077-68de7dec4831\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") "
Sep 11 22:09:49.856245 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.856099    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-workload-socket\") pod \"e959efbc-38af-44e0-9077-68de7dec4831\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") "
Sep 11 22:09:49.856245 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.856129    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e959efbc-38af-44e0-9077-68de7dec4831-istio-token\") pod \"e959efbc-38af-44e0-9077-68de7dec4831\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") "
Sep 11 22:09:49.856245 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.856150    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-workload-certs\") pod \"e959efbc-38af-44e0-9077-68de7dec4831\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") "
Sep 11 22:09:49.856245 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.856207    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-istio-envoy\") pod \"e959efbc-38af-44e0-9077-68de7dec4831\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") "
Sep 11 22:09:49.856494 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.856247    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-credential-socket\") pod \"e959efbc-38af-44e0-9077-68de7dec4831\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") "
Sep 11 22:09:49.856494 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.856290    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e959efbc-38af-44e0-9077-68de7dec4831-istiod-ca-cert\") pod \"e959efbc-38af-44e0-9077-68de7dec4831\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") "
Sep 11 22:09:49.856494 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.856344    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vrlz\" (UniqueName: \"kubernetes.io/projected/e959efbc-38af-44e0-9077-68de7dec4831-kube-api-access-5vrlz\") pod \"e959efbc-38af-44e0-9077-68de7dec4831\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") "
Sep 11 22:09:49.856494 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.856375    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-istio-data\") pod \"e959efbc-38af-44e0-9077-68de7dec4831\" (UID: \"e959efbc-38af-44e0-9077-68de7dec4831\") "
Sep 11 22:09:49.856710 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.856543    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e959efbc-38af-44e0-9077-68de7dec4831" (UID: "e959efbc-38af-44e0-9077-68de7dec4831"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:09:49.856710 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.856621    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e959efbc-38af-44e0-9077-68de7dec4831" (UID: "e959efbc-38af-44e0-9077-68de7dec4831"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:09:49.856821 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.856810    2587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-workload-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:09:49.856876 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.856829    2587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-workload-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:09:49.857277 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.857163    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-istio-data" (OuterVolumeSpecName: "istio-data") pod "e959efbc-38af-44e0-9077-68de7dec4831" (UID: "e959efbc-38af-44e0-9077-68de7dec4831"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:09:49.857277 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.857216    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e959efbc-38af-44e0-9077-68de7dec4831" (UID: "e959efbc-38af-44e0-9077-68de7dec4831"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:09:49.857480 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.857394    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e959efbc-38af-44e0-9077-68de7dec4831-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e959efbc-38af-44e0-9077-68de7dec4831" (UID: "e959efbc-38af-44e0-9077-68de7dec4831"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:09:49.858752 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.858722    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e959efbc-38af-44e0-9077-68de7dec4831" (UID: "e959efbc-38af-44e0-9077-68de7dec4831"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:09:49.858848 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.858823    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e959efbc-38af-44e0-9077-68de7dec4831-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e959efbc-38af-44e0-9077-68de7dec4831" (UID: "e959efbc-38af-44e0-9077-68de7dec4831"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:09:49.858957 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.858935    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e959efbc-38af-44e0-9077-68de7dec4831-kube-api-access-5vrlz" (OuterVolumeSpecName: "kube-api-access-5vrlz") pod "e959efbc-38af-44e0-9077-68de7dec4831" (UID: "e959efbc-38af-44e0-9077-68de7dec4831"). InnerVolumeSpecName "kube-api-access-5vrlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:09:49.859214 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.859198    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e959efbc-38af-44e0-9077-68de7dec4831-istio-token" (OuterVolumeSpecName: "istio-token") pod "e959efbc-38af-44e0-9077-68de7dec4831" (UID: "e959efbc-38af-44e0-9077-68de7dec4831"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:09:49.958282 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.958230    2587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-istio-envoy\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:09:49.958282 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.958274    2587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-credential-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:09:49.958282 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.958286    2587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e959efbc-38af-44e0-9077-68de7dec4831-istiod-ca-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:09:49.958282 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.958295    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vrlz\" (UniqueName: \"kubernetes.io/projected/e959efbc-38af-44e0-9077-68de7dec4831-kube-api-access-5vrlz\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:09:49.958282 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.958304    2587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e959efbc-38af-44e0-9077-68de7dec4831-istio-data\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:09:49.958617 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.958312    2587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e959efbc-38af-44e0-9077-68de7dec4831-istio-podinfo\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:09:49.958617 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:49.958320    2587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e959efbc-38af-44e0-9077-68de7dec4831-istio-token\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:09:50.814683 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:50.814657    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"
Sep 11 22:09:50.848821 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:50.848788    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"]
Sep 11 22:09:50.852222 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:50.852198    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-0aea78-openshift-default-556dd955b8-l8n89"]
Sep 11 22:09:51.342317 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:51.342279    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e959efbc-38af-44e0-9077-68de7dec4831" path="/var/lib/kubelet/pods/e959efbc-38af-44e0-9077-68de7dec4831/volumes"
Sep 11 22:09:53.294318 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.294287    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"]
Sep 11 22:09:53.294950 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.294927    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e959efbc-38af-44e0-9077-68de7dec4831" containerName="istio-proxy"
Sep 11 22:09:53.294950 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.294952    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="e959efbc-38af-44e0-9077-68de7dec4831" containerName="istio-proxy"
Sep 11 22:09:53.295094 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.294971    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e959efbc-38af-44e0-9077-68de7dec4831" containerName="istio-proxy"
Sep 11 22:09:53.295094 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.294979    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="e959efbc-38af-44e0-9077-68de7dec4831" containerName="istio-proxy"
Sep 11 22:09:53.295094 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.295072    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="e959efbc-38af-44e0-9077-68de7dec4831" containerName="istio-proxy"
Sep 11 22:09:53.299742 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.299718    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.302912 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.302891    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-5ef59c-openshift-default-dockercfg-lm4xd\""
Sep 11 22:09:53.323924 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.323891    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"]
Sep 11 22:09:53.390549 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.390513    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-workload-socket\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.390769 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.390592    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f777c7-66c9-4324-960d-7d8316053d1f-istiod-ca-cert\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.390769 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.390683    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-workload-certs\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.390769 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.390740    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8djw\" (UniqueName: \"kubernetes.io/projected/70f777c7-66c9-4324-960d-7d8316053d1f-kube-api-access-l8djw\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.390955 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.390788    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f777c7-66c9-4324-960d-7d8316053d1f-istio-podinfo\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.390955 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.390829    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-istio-envoy\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.390955 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.390861    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-credential-socket\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.390955 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.390911    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f777c7-66c9-4324-960d-7d8316053d1f-istio-token\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.390955 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.390945    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-istio-data\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.492196 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.492154    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-workload-socket\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.492196 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.492210    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f777c7-66c9-4324-960d-7d8316053d1f-istiod-ca-cert\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.492464 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.492260    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-workload-certs\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.492464 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.492285    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8djw\" (UniqueName: \"kubernetes.io/projected/70f777c7-66c9-4324-960d-7d8316053d1f-kube-api-access-l8djw\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.492464 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.492318    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f777c7-66c9-4324-960d-7d8316053d1f-istio-podinfo\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.492464 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.492374    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-istio-envoy\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.492464 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.492405    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-credential-socket\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.492464 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.492438    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f777c7-66c9-4324-960d-7d8316053d1f-istio-token\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.492730 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.492467    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-istio-data\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.492730 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.492666    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-workload-certs\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.492730 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.492678    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-workload-socket\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.492988 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.492961    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-istio-data\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.493156 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.493132    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-credential-socket\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.493203 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.493147    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f777c7-66c9-4324-960d-7d8316053d1f-istiod-ca-cert\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.495358 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.495297    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-istio-envoy\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.495548 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.495518    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f777c7-66c9-4324-960d-7d8316053d1f-istio-podinfo\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.500707 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.500675    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8djw\" (UniqueName: \"kubernetes.io/projected/70f777c7-66c9-4324-960d-7d8316053d1f-kube-api-access-l8djw\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.501110 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.501089    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f777c7-66c9-4324-960d-7d8316053d1f-istio-token\") pod \"e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") " pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.610396 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.610302    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:53.890024 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.889996    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"]
Sep 11 22:09:53.891681 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:09:53.891646    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f777c7_66c9_4324_960d_7d8316053d1f.slice/crio-abd2b169aedc25459df563e1d93eea14edcbe9c6fead4f21d969542a98d79bf9 WatchSource:0}: Error finding container abd2b169aedc25459df563e1d93eea14edcbe9c6fead4f21d969542a98d79bf9: Status 404 returned error can't find the container with id abd2b169aedc25459df563e1d93eea14edcbe9c6fead4f21d969542a98d79bf9
Sep 11 22:09:53.894097 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.894066    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:09:53.894189 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.894127    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:09:53.894189 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:53.894155    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:09:54.839144 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:54.839104    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk" event={"ID":"70f777c7-66c9-4324-960d-7d8316053d1f","Type":"ContainerStarted","Data":"41463a052eeed6d1ebd1bddf4f7eeb90934712f0cf77e84598ec95dc6cb0ec20"}
Sep 11 22:09:54.839557 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:54.839171    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk" event={"ID":"70f777c7-66c9-4324-960d-7d8316053d1f","Type":"ContainerStarted","Data":"abd2b169aedc25459df563e1d93eea14edcbe9c6fead4f21d969542a98d79bf9"}
Sep 11 22:09:54.863345 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:54.863278    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk" podStartSLOduration=1.863260793 podStartE2EDuration="1.863260793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:09:54.860856399 +0000 UTC m=+1048.035932779" watchObservedRunningTime="2026-09-11 22:09:54.863260793 +0000 UTC m=+1048.038337162"
Sep 11 22:09:55.610847 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:55.610803    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:55.616345 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:55.616294    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:55.847191 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:55.847159    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:55.848414 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:55.848392    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:09:56.576223 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:09:56.576176    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:10:20.992599 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:20.992562    2587 generic.go:356] "Generic (PLEG): container finished" podID="70f777c7-66c9-4324-960d-7d8316053d1f" containerID="41463a052eeed6d1ebd1bddf4f7eeb90934712f0cf77e84598ec95dc6cb0ec20" exitCode=0
Sep 11 22:10:20.993045 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:20.992605    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk" event={"ID":"70f777c7-66c9-4324-960d-7d8316053d1f","Type":"ContainerDied","Data":"41463a052eeed6d1ebd1bddf4f7eeb90934712f0cf77e84598ec95dc6cb0ec20"}
Sep 11 22:10:20.993045 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:20.993011    2587 scope.go:117] "RemoveContainer" containerID="41463a052eeed6d1ebd1bddf4f7eeb90934712f0cf77e84598ec95dc6cb0ec20"
Sep 11 22:10:20.994255 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:20.994227    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:10:20.994318 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:20.994295    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:10:20.994379 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:20.994339    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:10:22.000154 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.000115    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk" event={"ID":"70f777c7-66c9-4324-960d-7d8316053d1f","Type":"ContainerStarted","Data":"b9cc14caeb25bf621c23054689181f6f40f234fb307f31e6c71018ef333f4f66"}
Sep 11 22:10:22.315615 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.315526    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"]
Sep 11 22:10:22.316340 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.316306    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="e959efbc-38af-44e0-9077-68de7dec4831" containerName="istio-proxy"
Sep 11 22:10:22.319907 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.319877    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.322836 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.322813    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-38f772-openshift-default-dockercfg-bscvz\""
Sep 11 22:10:22.331269 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.331239    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"]
Sep 11 22:10:22.493590 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.493556    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-istio-data\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.493784 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.493600    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07be57ee-7747-4608-820d-b20afeaf3bd9-istio-podinfo\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.493784 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.493631    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-istio-envoy\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.493784 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.493695    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07be57ee-7747-4608-820d-b20afeaf3bd9-istio-token\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.493784 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.493759    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-workload-socket\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.493978 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.493799    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-workload-certs\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.493978 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.493818    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-credential-socket\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.493978 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.493957    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9c6v\" (UniqueName: \"kubernetes.io/projected/07be57ee-7747-4608-820d-b20afeaf3bd9-kube-api-access-n9c6v\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.494103 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.493997    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07be57ee-7747-4608-820d-b20afeaf3bd9-istiod-ca-cert\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.594724 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.594645    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-workload-certs\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.594724 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.594684    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-credential-socket\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.594956 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.594739    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9c6v\" (UniqueName: \"kubernetes.io/projected/07be57ee-7747-4608-820d-b20afeaf3bd9-kube-api-access-n9c6v\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.594956 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.594755    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07be57ee-7747-4608-820d-b20afeaf3bd9-istiod-ca-cert\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.594956 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.594781    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-istio-data\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.595133 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.595034    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-workload-certs\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.595133 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.595096    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07be57ee-7747-4608-820d-b20afeaf3bd9-istio-podinfo\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.595238 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.595134    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-istio-data\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.595238 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.595152    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-istio-envoy\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.595238 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.595181    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07be57ee-7747-4608-820d-b20afeaf3bd9-istio-token\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.595385 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.595241    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-workload-socket\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.595522 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.595490    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-credential-socket\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.595724 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.595706    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-workload-socket\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.595859 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.595841    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07be57ee-7747-4608-820d-b20afeaf3bd9-istiod-ca-cert\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.597676 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.597654    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-istio-envoy\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.597758 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.597733    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07be57ee-7747-4608-820d-b20afeaf3bd9-istio-podinfo\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.603871 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.603843    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9c6v\" (UniqueName: \"kubernetes.io/projected/07be57ee-7747-4608-820d-b20afeaf3bd9-kube-api-access-n9c6v\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.603999 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.603981    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07be57ee-7747-4608-820d-b20afeaf3bd9-istio-token\") pod \"e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") " pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.610596 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.610576    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:10:22.611595 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.611575    2587 patch_prober.go:28] interesting pod/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 11 22:10:22.611667 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.611633    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk" podUID="70f777c7-66c9-4324-960d-7d8316053d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 11 22:10:22.634387 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.634347    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:22.935593 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.933400    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"]
Sep 11 22:10:22.944034 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.943993    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:10:22.944150 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.944084    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:10:22.944150 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:22.944130    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:10:23.009922 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:23.009891    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2" event={"ID":"07be57ee-7747-4608-820d-b20afeaf3bd9","Type":"ContainerStarted","Data":"76eac1399a749e4ca8aca90162da935c44b588796522877a17774e240a3bd276"}
Sep 11 22:10:23.611356 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:23.611310    2587 patch_prober.go:28] interesting pod/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 11 22:10:23.611517 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:23.611370    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk" podUID="70f777c7-66c9-4324-960d-7d8316053d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 11 22:10:24.016988 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:24.016949    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2" event={"ID":"07be57ee-7747-4608-820d-b20afeaf3bd9","Type":"ContainerStarted","Data":"d98cd1e47b35733ce05bff04f95285987c5ce2184ddd8b736e97353fd30bed2b"}
Sep 11 22:10:24.046865 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:24.046785    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2" podStartSLOduration=2.046761704 podStartE2EDuration="2.046761704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:10:24.042871417 +0000 UTC m=+1077.217947802" watchObservedRunningTime="2026-09-11 22:10:24.046761704 +0000 UTC m=+1077.221838076"
Sep 11 22:10:24.610991 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:24.610948    2587 patch_prober.go:28] interesting pod/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 11 22:10:24.611197 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:24.611009    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk" podUID="70f777c7-66c9-4324-960d-7d8316053d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 11 22:10:24.635159 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:24.635122    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:24.646963 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:24.646931    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:25.026558 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:25.026530    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:25.028657 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:25.028632    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:25.087747 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:25.087710    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:10:25.611593 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:25.611555    2587 patch_prober.go:28] interesting pod/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 11 22:10:25.611780 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:25.611610    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk" podUID="70f777c7-66c9-4324-960d-7d8316053d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 11 22:10:25.848001 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:25.847959    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:10:26.611258 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:26.611218    2587 patch_prober.go:28] interesting pod/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 11 22:10:26.611672 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:26.611287    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk" podUID="70f777c7-66c9-4324-960d-7d8316053d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 11 22:10:27.611211 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:27.611173    2587 patch_prober.go:28] interesting pod/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 11 22:10:27.611399 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:27.611244    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk" podUID="70f777c7-66c9-4324-960d-7d8316053d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 11 22:10:28.615138 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:28.615103    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:10:28.616042 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:28.616024    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:10:41.419525 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:41.419488    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7681-753fb8"]
Sep 11 22:10:41.425033 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:41.424999    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-753fb8"
Sep 11 22:10:41.427932 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:41.427904    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h4c6b\""
Sep 11 22:10:41.434284 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:41.434258    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-753fb8"]
Sep 11 22:10:41.492307 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:41.492261    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m56cp\" (UniqueName: \"kubernetes.io/projected/7c61f95d-9889-4440-ac20-9b7dce1faf7c-kube-api-access-m56cp\") pod \"test-curl-7681-753fb8\" (UID: \"7c61f95d-9889-4440-ac20-9b7dce1faf7c\") " pod="opendatahub/test-curl-7681-753fb8"
Sep 11 22:10:41.593602 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:41.593560    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m56cp\" (UniqueName: \"kubernetes.io/projected/7c61f95d-9889-4440-ac20-9b7dce1faf7c-kube-api-access-m56cp\") pod \"test-curl-7681-753fb8\" (UID: \"7c61f95d-9889-4440-ac20-9b7dce1faf7c\") " pod="opendatahub/test-curl-7681-753fb8"
Sep 11 22:10:41.601743 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:41.601713    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m56cp\" (UniqueName: \"kubernetes.io/projected/7c61f95d-9889-4440-ac20-9b7dce1faf7c-kube-api-access-m56cp\") pod \"test-curl-7681-753fb8\" (UID: \"7c61f95d-9889-4440-ac20-9b7dce1faf7c\") " pod="opendatahub/test-curl-7681-753fb8"
Sep 11 22:10:41.741537 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:41.741430    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-753fb8"
Sep 11 22:10:42.015072 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:42.015041    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-753fb8"]
Sep 11 22:10:42.016844 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:10:42.016822    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c61f95d_9889_4440_ac20_9b7dce1faf7c.slice/crio-2c05dafcf79d1534c787e51d136cca657e5b7a2396b4afbdea70df09cd79b038 WatchSource:0}: Error finding container 2c05dafcf79d1534c787e51d136cca657e5b7a2396b4afbdea70df09cd79b038: Status 404 returned error can't find the container with id 2c05dafcf79d1534c787e51d136cca657e5b7a2396b4afbdea70df09cd79b038
Sep 11 22:10:42.018928 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:42.018912    2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 22:10:42.150920 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:42.150879    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-753fb8" event={"ID":"7c61f95d-9889-4440-ac20-9b7dce1faf7c","Type":"ContainerStarted","Data":"2c05dafcf79d1534c787e51d136cca657e5b7a2396b4afbdea70df09cd79b038"}
Sep 11 22:10:44.757228 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:44.757189    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz"]
Sep 11 22:10:44.757694 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:44.757437    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz" podUID="9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf" containerName="maas-api" containerID="cri-o://18144a86bed2402eea8707ecc7f330f9b61adcacc35b67e29f35dea5ae42a8b4" gracePeriod=30
Sep 11 22:10:45.050377 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.050041    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz"
Sep 11 22:10:45.129077 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.129027    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r78t5\" (UniqueName: \"kubernetes.io/projected/9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf-kube-api-access-r78t5\") pod \"9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf\" (UID: \"9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf\") "
Sep 11 22:10:45.129273 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.129085    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf-openshift-service-ca\") pod \"9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf\" (UID: \"9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf\") "
Sep 11 22:10:45.129361 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.129276    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf-maas-api-tls\") pod \"9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf\" (UID: \"9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf\") "
Sep 11 22:10:45.129429 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.129317    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf-metrics-certs\") pod \"9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf\" (UID: \"9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf\") "
Sep 11 22:10:45.129839 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.129629    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf" (UID: "9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:10:45.129839 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.129733    2587 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf-openshift-service-ca\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:10:45.132317 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.132287    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf-kube-api-access-r78t5" (OuterVolumeSpecName: "kube-api-access-r78t5") pod "9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf" (UID: "9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf"). InnerVolumeSpecName "kube-api-access-r78t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:10:45.132525 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.132484    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf" (UID: "9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:10:45.132717 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.132679    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf" (UID: "9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:10:45.173612 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.173578    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-753fb8" event={"ID":"7c61f95d-9889-4440-ac20-9b7dce1faf7c","Type":"ContainerStarted","Data":"8cc933861ea4bbc82f4dbcda8a794a63fa133a42f54287605e1adca47894884a"}
Sep 11 22:10:45.175064 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.175033    2587 generic.go:356] "Generic (PLEG): container finished" podID="9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf" containerID="18144a86bed2402eea8707ecc7f330f9b61adcacc35b67e29f35dea5ae42a8b4" exitCode=0
Sep 11 22:10:45.175183 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.175065    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz"
Sep 11 22:10:45.175183 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.175078    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz" event={"ID":"9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf","Type":"ContainerDied","Data":"18144a86bed2402eea8707ecc7f330f9b61adcacc35b67e29f35dea5ae42a8b4"}
Sep 11 22:10:45.175183 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.175127    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz" event={"ID":"9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf","Type":"ContainerDied","Data":"898acfe967a4254063e864af858e531c4950fe2e35562fa7e59cc7fcb810a86e"}
Sep 11 22:10:45.175183 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.175157    2587 scope.go:117] "RemoveContainer" containerID="18144a86bed2402eea8707ecc7f330f9b61adcacc35b67e29f35dea5ae42a8b4"
Sep 11 22:10:45.188292 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.188261    2587 scope.go:117] "RemoveContainer" containerID="18144a86bed2402eea8707ecc7f330f9b61adcacc35b67e29f35dea5ae42a8b4"
Sep 11 22:10:45.188673 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:45.188652    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18144a86bed2402eea8707ecc7f330f9b61adcacc35b67e29f35dea5ae42a8b4\": container with ID starting with 18144a86bed2402eea8707ecc7f330f9b61adcacc35b67e29f35dea5ae42a8b4 not found: ID does not exist" containerID="18144a86bed2402eea8707ecc7f330f9b61adcacc35b67e29f35dea5ae42a8b4"
Sep 11 22:10:45.188758 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.188682    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18144a86bed2402eea8707ecc7f330f9b61adcacc35b67e29f35dea5ae42a8b4"} err="failed to get container status \"18144a86bed2402eea8707ecc7f330f9b61adcacc35b67e29f35dea5ae42a8b4\": rpc error: code = NotFound desc = could not find container \"18144a86bed2402eea8707ecc7f330f9b61adcacc35b67e29f35dea5ae42a8b4\": container with ID starting with 18144a86bed2402eea8707ecc7f330f9b61adcacc35b67e29f35dea5ae42a8b4 not found: ID does not exist"
Sep 11 22:10:45.191272 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.191180    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7681-753fb8" podStartSLOduration=1.544055502 podStartE2EDuration="4.191158742s" totalImagesPullingTime="2.64710324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:10:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:10:45.188014477 +0000 UTC m=+1098.363090861" watchObservedRunningTime="2026-09-11 22:10:45.191158742 +0000 UTC m=+1098.366235114"
Sep 11 22:10:45.201519 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.201488    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz"]
Sep 11 22:10:45.205674 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.205644    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-27dad9-64455795b9-5tbbz"]
Sep 11 22:10:45.230826 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.230794    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf-maas-api-tls\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:10:45.230826 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.230823    2587 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf-metrics-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:10:45.231003 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.230836    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r78t5\" (UniqueName: \"kubernetes.io/projected/9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf-kube-api-access-r78t5\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:10:45.338552 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.338511    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf" path="/var/lib/kubelet/pods/9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf/volumes"
Sep 11 22:10:45.848375 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.848321    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7681-753fb8"]
Sep 11 22:10:45.853540 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.853504    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7681-753fb8"]
Sep 11 22:10:45.967394 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.967360    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7681-dfa50e"]
Sep 11 22:10:45.967839 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.967824    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf" containerName="maas-api"
Sep 11 22:10:45.967839 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.967838    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf" containerName="maas-api"
Sep 11 22:10:45.968019 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.967854    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c61f95d-9889-4440-ac20-9b7dce1faf7c" containerName="test-curl-7681-753fb8"
Sep 11 22:10:45.968019 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.967859    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c61f95d-9889-4440-ac20-9b7dce1faf7c" containerName="test-curl-7681-753fb8"
Sep 11 22:10:45.968019 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.967932    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c61f95d-9889-4440-ac20-9b7dce1faf7c" containerName="test-curl-7681-753fb8"
Sep 11 22:10:45.968019 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.967940    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d21f55a-2bfb-4ffa-baf0-d3eb359b67bf" containerName="maas-api"
Sep 11 22:10:45.972637 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.972607    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-dfa50e"
Sep 11 22:10:45.978842 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:45.978815    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-dfa50e"]
Sep 11 22:10:46.041223 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:46.041181    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qpl8\" (UniqueName: \"kubernetes.io/projected/33998f16-e1aa-4e8f-adb0-e5e767c2cb24-kube-api-access-4qpl8\") pod \"test-curl-7681-dfa50e\" (UID: \"33998f16-e1aa-4e8f-adb0-e5e767c2cb24\") " pod="opendatahub/test-curl-7681-dfa50e"
Sep 11 22:10:46.141982 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:46.141944    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qpl8\" (UniqueName: \"kubernetes.io/projected/33998f16-e1aa-4e8f-adb0-e5e767c2cb24-kube-api-access-4qpl8\") pod \"test-curl-7681-dfa50e\" (UID: \"33998f16-e1aa-4e8f-adb0-e5e767c2cb24\") " pod="opendatahub/test-curl-7681-dfa50e"
Sep 11 22:10:46.150065 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:46.150033    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qpl8\" (UniqueName: \"kubernetes.io/projected/33998f16-e1aa-4e8f-adb0-e5e767c2cb24-kube-api-access-4qpl8\") pod \"test-curl-7681-dfa50e\" (UID: \"33998f16-e1aa-4e8f-adb0-e5e767c2cb24\") " pod="opendatahub/test-curl-7681-dfa50e"
Sep 11 22:10:46.181441 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:46.181408    2587 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7681-753fb8" container="test-curl-7681-753fb8"
Sep 11 22:10:46.287283 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:46.287205    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-dfa50e"
Sep 11 22:10:46.568276 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:46.568248    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-dfa50e"]
Sep 11 22:10:46.569659 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:10:46.569623    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33998f16_e1aa_4e8f_adb0_e5e767c2cb24.slice/crio-8808ce24dbb796818fc8da0947a8730550925303d35939efdcf30be98786cb54 WatchSource:0}: Error finding container 8808ce24dbb796818fc8da0947a8730550925303d35939efdcf30be98786cb54: Status 404 returned error can't find the container with id 8808ce24dbb796818fc8da0947a8730550925303d35939efdcf30be98786cb54
Sep 11 22:10:47.188468 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:47.188430    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-dfa50e" event={"ID":"33998f16-e1aa-4e8f-adb0-e5e767c2cb24","Type":"ContainerStarted","Data":"62904f76cb6770a412bafb37c9b02b8a69b932b7c73ff921e12c570c320b0e4f"}
Sep 11 22:10:47.188932 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:47.188481    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-dfa50e" event={"ID":"33998f16-e1aa-4e8f-adb0-e5e767c2cb24","Type":"ContainerStarted","Data":"8808ce24dbb796818fc8da0947a8730550925303d35939efdcf30be98786cb54"}
Sep 11 22:10:47.190021 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:47.189997    2587 generic.go:356] "Generic (PLEG): container finished" podID="07be57ee-7747-4608-820d-b20afeaf3bd9" containerID="d98cd1e47b35733ce05bff04f95285987c5ce2184ddd8b736e97353fd30bed2b" exitCode=0
Sep 11 22:10:47.190144 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:47.190069    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2" event={"ID":"07be57ee-7747-4608-820d-b20afeaf3bd9","Type":"ContainerDied","Data":"d98cd1e47b35733ce05bff04f95285987c5ce2184ddd8b736e97353fd30bed2b"}
Sep 11 22:10:47.190211 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:47.190183    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7681-753fb8" podUID="7c61f95d-9889-4440-ac20-9b7dce1faf7c" containerName="test-curl-7681-753fb8" containerID="cri-o://8cc933861ea4bbc82f4dbcda8a794a63fa133a42f54287605e1adca47894884a" gracePeriod=2
Sep 11 22:10:47.190569 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:47.190486    2587 scope.go:117] "RemoveContainer" containerID="d98cd1e47b35733ce05bff04f95285987c5ce2184ddd8b736e97353fd30bed2b"
Sep 11 22:10:47.191757 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:47.191721    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:10:47.191837 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:47.191815    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:10:47.191891 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:47.191867    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:10:47.203240 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:47.203179    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7681-dfa50e" podStartSLOduration=2.203164239 podStartE2EDuration="2.203164239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:10:47.203128915 +0000 UTC m=+1100.378205286" watchObservedRunningTime="2026-09-11 22:10:47.203164239 +0000 UTC m=+1100.378240609"
Sep 11 22:10:47.226835 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:47.226810    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-753fb8\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="7c61f95d-9889-4440-ac20-9b7dce1faf7c" pod="opendatahub/test-curl-7681-753fb8"
Sep 11 22:10:47.335001 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:47.334672    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-753fb8\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="7c61f95d-9889-4440-ac20-9b7dce1faf7c" pod="opendatahub/test-curl-7681-753fb8"
Sep 11 22:10:47.811400 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:47.811369    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7681-dfa50e"]
Sep 11 22:10:47.814665 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:47.814631    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7681-dfa50e"]
Sep 11 22:10:47.940352 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:47.935493    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7681-0f3f2d"]
Sep 11 22:10:47.940352 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:47.936195    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33998f16-e1aa-4e8f-adb0-e5e767c2cb24" containerName="test-curl-7681-dfa50e"
Sep 11 22:10:47.940352 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:47.936215    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="33998f16-e1aa-4e8f-adb0-e5e767c2cb24" containerName="test-curl-7681-dfa50e"
Sep 11 22:10:47.940352 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:47.936353    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="33998f16-e1aa-4e8f-adb0-e5e767c2cb24" containerName="test-curl-7681-dfa50e"
Sep 11 22:10:47.948833 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:47.948794    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-0f3f2d"]
Sep 11 22:10:47.949005 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:47.948893    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-0f3f2d"
Sep 11 22:10:48.063309 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:48.063254    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsgf8\" (UniqueName: \"kubernetes.io/projected/4f79a3e6-94c3-4b42-972e-ea89018c54f3-kube-api-access-gsgf8\") pod \"test-curl-7681-0f3f2d\" (UID: \"4f79a3e6-94c3-4b42-972e-ea89018c54f3\") " pod="opendatahub/test-curl-7681-0f3f2d"
Sep 11 22:10:48.164760 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:48.164725    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsgf8\" (UniqueName: \"kubernetes.io/projected/4f79a3e6-94c3-4b42-972e-ea89018c54f3-kube-api-access-gsgf8\") pod \"test-curl-7681-0f3f2d\" (UID: \"4f79a3e6-94c3-4b42-972e-ea89018c54f3\") " pod="opendatahub/test-curl-7681-0f3f2d"
Sep 11 22:10:48.172650 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:48.172622    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsgf8\" (UniqueName: \"kubernetes.io/projected/4f79a3e6-94c3-4b42-972e-ea89018c54f3-kube-api-access-gsgf8\") pod \"test-curl-7681-0f3f2d\" (UID: \"4f79a3e6-94c3-4b42-972e-ea89018c54f3\") " pod="opendatahub/test-curl-7681-0f3f2d"
Sep 11 22:10:48.198027 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:48.197944    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2" event={"ID":"07be57ee-7747-4608-820d-b20afeaf3bd9","Type":"ContainerStarted","Data":"009726f2597f48ac17ed6b324b4853cf2dbd5c66d6e594c7d1fa1f7686675130"}
Sep 11 22:10:48.198402 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:48.198171    2587 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7681-dfa50e" container="test-curl-7681-dfa50e"
Sep 11 22:10:48.263006 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:48.262968    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-0f3f2d"
Sep 11 22:10:48.538431 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:48.538387    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-0f3f2d"]
Sep 11 22:10:48.635270 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:48.635053    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:48.635270 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:48.635172    2587 patch_prober.go:28] interesting pod/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 11 22:10:48.635270 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:48.635211    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2" podUID="07be57ee-7747-4608-820d-b20afeaf3bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 11 22:10:49.206194 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:49.205989    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-0f3f2d" event={"ID":"4f79a3e6-94c3-4b42-972e-ea89018c54f3","Type":"ContainerStarted","Data":"7eaef6f2a98dec4f49727ce37a9030f310d4a7395af47e0f3936a43d2bf68779"}
Sep 11 22:10:49.206194 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:49.206034    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-0f3f2d" event={"ID":"4f79a3e6-94c3-4b42-972e-ea89018c54f3","Type":"ContainerStarted","Data":"b993d6112c7936b3edbe435daf873fea49d86d3e4109a9dbe882176981f5907b"}
Sep 11 22:10:49.206612 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:49.206043    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7681-dfa50e" podUID="33998f16-e1aa-4e8f-adb0-e5e767c2cb24" containerName="test-curl-7681-dfa50e" containerID="cri-o://62904f76cb6770a412bafb37c9b02b8a69b932b7c73ff921e12c570c320b0e4f" gracePeriod=2
Sep 11 22:10:49.208756 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:49.208729    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-dfa50e\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="33998f16-e1aa-4e8f-adb0-e5e767c2cb24" pod="opendatahub/test-curl-7681-dfa50e"
Sep 11 22:10:49.227896 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:49.227838    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-dfa50e\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="33998f16-e1aa-4e8f-adb0-e5e767c2cb24" pod="opendatahub/test-curl-7681-dfa50e"
Sep 11 22:10:49.448557 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:49.448469    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-753fb8"
Sep 11 22:10:49.479223 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:49.479193    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m56cp\" (UniqueName: \"kubernetes.io/projected/7c61f95d-9889-4440-ac20-9b7dce1faf7c-kube-api-access-m56cp\") pod \"7c61f95d-9889-4440-ac20-9b7dce1faf7c\" (UID: \"7c61f95d-9889-4440-ac20-9b7dce1faf7c\") "
Sep 11 22:10:49.481165 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:49.481135    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c61f95d-9889-4440-ac20-9b7dce1faf7c-kube-api-access-m56cp" (OuterVolumeSpecName: "kube-api-access-m56cp") pod "7c61f95d-9889-4440-ac20-9b7dce1faf7c" (UID: "7c61f95d-9889-4440-ac20-9b7dce1faf7c"). InnerVolumeSpecName "kube-api-access-m56cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:10:49.579942 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:49.579899    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m56cp\" (UniqueName: \"kubernetes.io/projected/7c61f95d-9889-4440-ac20-9b7dce1faf7c-kube-api-access-m56cp\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:10:49.635706 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:49.635660    2587 patch_prober.go:28] interesting pod/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 11 22:10:49.635918 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:49.635739    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2" podUID="07be57ee-7747-4608-820d-b20afeaf3bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 11 22:10:49.847626 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:49.847487    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7681-0f3f2d" podStartSLOduration=2.847465021 podStartE2EDuration="2.847465021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:10:49.225495676 +0000 UTC m=+1102.400572047" watchObservedRunningTime="2026-09-11 22:10:49.847465021 +0000 UTC m=+1103.022541395"
Sep 11 22:10:49.850052 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:49.850018    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7681-0f3f2d"]
Sep 11 22:10:49.853063 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:49.853041    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7681-0f3f2d"]
Sep 11 22:10:49.965171 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:49.965087    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7681-4c9986"]
Sep 11 22:10:49.965631 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:49.965612    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f79a3e6-94c3-4b42-972e-ea89018c54f3" containerName="test-curl-7681-0f3f2d"
Sep 11 22:10:49.965631 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:49.965633    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f79a3e6-94c3-4b42-972e-ea89018c54f3" containerName="test-curl-7681-0f3f2d"
Sep 11 22:10:49.965743 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:49.965727    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f79a3e6-94c3-4b42-972e-ea89018c54f3" containerName="test-curl-7681-0f3f2d"
Sep 11 22:10:49.969057 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:49.969035    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-4c9986"
Sep 11 22:10:49.975527 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:49.975500    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-4c9986"]
Sep 11 22:10:50.084150 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:50.084107    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4q9g\" (UniqueName: \"kubernetes.io/projected/b9b42bb2-41ad-4e9e-b41b-835186d66ed3-kube-api-access-l4q9g\") pod \"test-curl-7681-4c9986\" (UID: \"b9b42bb2-41ad-4e9e-b41b-835186d66ed3\") " pod="opendatahub/test-curl-7681-4c9986"
Sep 11 22:10:50.185414 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:50.185369    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4q9g\" (UniqueName: \"kubernetes.io/projected/b9b42bb2-41ad-4e9e-b41b-835186d66ed3-kube-api-access-l4q9g\") pod \"test-curl-7681-4c9986\" (UID: \"b9b42bb2-41ad-4e9e-b41b-835186d66ed3\") " pod="opendatahub/test-curl-7681-4c9986"
Sep 11 22:10:50.193779 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:50.193742    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4q9g\" (UniqueName: \"kubernetes.io/projected/b9b42bb2-41ad-4e9e-b41b-835186d66ed3-kube-api-access-l4q9g\") pod \"test-curl-7681-4c9986\" (UID: \"b9b42bb2-41ad-4e9e-b41b-835186d66ed3\") " pod="opendatahub/test-curl-7681-4c9986"
Sep 11 22:10:50.212814 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:50.212776    2587 generic.go:356] "Generic (PLEG): container finished" podID="7c61f95d-9889-4440-ac20-9b7dce1faf7c" containerID="8cc933861ea4bbc82f4dbcda8a794a63fa133a42f54287605e1adca47894884a" exitCode=137
Sep 11 22:10:50.212814 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:50.212816    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-753fb8"
Sep 11 22:10:50.213344 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:50.212874    2587 scope.go:117] "RemoveContainer" containerID="8cc933861ea4bbc82f4dbcda8a794a63fa133a42f54287605e1adca47894884a"
Sep 11 22:10:50.213344 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:50.213131    2587 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7681-0f3f2d" container="test-curl-7681-0f3f2d"
Sep 11 22:10:50.233847 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:50.233257    2587 scope.go:117] "RemoveContainer" containerID="8cc933861ea4bbc82f4dbcda8a794a63fa133a42f54287605e1adca47894884a"
Sep 11 22:10:50.233847 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:50.233740    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc933861ea4bbc82f4dbcda8a794a63fa133a42f54287605e1adca47894884a\": container with ID starting with 8cc933861ea4bbc82f4dbcda8a794a63fa133a42f54287605e1adca47894884a not found: ID does not exist" containerID="8cc933861ea4bbc82f4dbcda8a794a63fa133a42f54287605e1adca47894884a"
Sep 11 22:10:50.233847 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:50.233781    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc933861ea4bbc82f4dbcda8a794a63fa133a42f54287605e1adca47894884a"} err="failed to get container status \"8cc933861ea4bbc82f4dbcda8a794a63fa133a42f54287605e1adca47894884a\": rpc error: code = NotFound desc = could not find container \"8cc933861ea4bbc82f4dbcda8a794a63fa133a42f54287605e1adca47894884a\": container with ID starting with 8cc933861ea4bbc82f4dbcda8a794a63fa133a42f54287605e1adca47894884a not found: ID does not exist"
Sep 11 22:10:50.239442 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:50.239401    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-753fb8\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="7c61f95d-9889-4440-ac20-9b7dce1faf7c" pod="opendatahub/test-curl-7681-753fb8"
Sep 11 22:10:50.280866 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:50.280831    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-4c9986"
Sep 11 22:10:50.585603 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:50.585571    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7681-4c9986"]
Sep 11 22:10:50.587557 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:10:50.587516    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9b42bb2_41ad_4e9e_b41b_835186d66ed3.slice/crio-178c42b7c0e45e22c0548e44273086810e7123db4babaaba58a72e34ee763dd9 WatchSource:0}: Error finding container 178c42b7c0e45e22c0548e44273086810e7123db4babaaba58a72e34ee763dd9: Status 404 returned error can't find the container with id 178c42b7c0e45e22c0548e44273086810e7123db4babaaba58a72e34ee763dd9
Sep 11 22:10:50.637353 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:50.635400    2587 patch_prober.go:28] interesting pod/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 11 22:10:50.637353 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:50.635463    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2" podUID="07be57ee-7747-4608-820d-b20afeaf3bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 11 22:10:51.223715 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:51.222499    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7681-0f3f2d" podUID="4f79a3e6-94c3-4b42-972e-ea89018c54f3" containerName="test-curl-7681-0f3f2d" containerID="cri-o://7eaef6f2a98dec4f49727ce37a9030f310d4a7395af47e0f3936a43d2bf68779" gracePeriod=2
Sep 11 22:10:51.223715 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:51.223484    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-4c9986" event={"ID":"b9b42bb2-41ad-4e9e-b41b-835186d66ed3","Type":"ContainerStarted","Data":"4fba854fc800af89883d77f5d3ef29df9b49bcce43ccdec2ef9ff65c98fc8040"}
Sep 11 22:10:51.223715 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:51.223521    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7681-4c9986" event={"ID":"b9b42bb2-41ad-4e9e-b41b-835186d66ed3","Type":"ContainerStarted","Data":"178c42b7c0e45e22c0548e44273086810e7123db4babaaba58a72e34ee763dd9"}
Sep 11 22:10:51.246770 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:51.246710    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7681-4c9986" podStartSLOduration=2.246675137 podStartE2EDuration="2.246675137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:10:51.245379191 +0000 UTC m=+1104.420455562" watchObservedRunningTime="2026-09-11 22:10:51.246675137 +0000 UTC m=+1104.421751510"
Sep 11 22:10:51.248107 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:51.247960    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-0f3f2d\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="4f79a3e6-94c3-4b42-972e-ea89018c54f3" pod="opendatahub/test-curl-7681-0f3f2d"
Sep 11 22:10:51.250742 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:51.250571    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-753fb8\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="7c61f95d-9889-4440-ac20-9b7dce1faf7c" pod="opendatahub/test-curl-7681-753fb8"
Sep 11 22:10:51.252836 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:51.252810    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-0f3f2d\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="4f79a3e6-94c3-4b42-972e-ea89018c54f3" pod="opendatahub/test-curl-7681-0f3f2d"
Sep 11 22:10:51.255071 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:51.255039    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-753fb8\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="7c61f95d-9889-4440-ac20-9b7dce1faf7c" pod="opendatahub/test-curl-7681-753fb8"
Sep 11 22:10:51.262729 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:51.262539    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"]
Sep 11 22:10:51.266433 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:51.262827    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq" podUID="f2f9ee75-e2ab-45e6-9f74-9088f1508251" containerName="istio-proxy" containerID="cri-o://1063e304cdbd52beb4e3c869d51be44ceedf903bea53f7e83f14d679a293e932" gracePeriod=30
Sep 11 22:10:51.266433 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:51.265071    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-753fb8\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="7c61f95d-9889-4440-ac20-9b7dce1faf7c" pod="opendatahub/test-curl-7681-753fb8"
Sep 11 22:10:51.294179 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:51.294135    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-0f3f2d\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="4f79a3e6-94c3-4b42-972e-ea89018c54f3" pod="opendatahub/test-curl-7681-0f3f2d"
Sep 11 22:10:51.302244 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:51.302213    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:10:51.354257 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:51.352623    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c61f95d-9889-4440-ac20-9b7dce1faf7c" path="/var/lib/kubelet/pods/7c61f95d-9889-4440-ac20-9b7dce1faf7c/volumes"
Sep 11 22:10:51.458519 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:51.458486    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-dfa50e"
Sep 11 22:10:51.499082 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:51.499017    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qpl8\" (UniqueName: \"kubernetes.io/projected/33998f16-e1aa-4e8f-adb0-e5e767c2cb24-kube-api-access-4qpl8\") pod \"33998f16-e1aa-4e8f-adb0-e5e767c2cb24\" (UID: \"33998f16-e1aa-4e8f-adb0-e5e767c2cb24\") "
Sep 11 22:10:51.500928 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:51.500901    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33998f16-e1aa-4e8f-adb0-e5e767c2cb24-kube-api-access-4qpl8" (OuterVolumeSpecName: "kube-api-access-4qpl8") pod "33998f16-e1aa-4e8f-adb0-e5e767c2cb24" (UID: "33998f16-e1aa-4e8f-adb0-e5e767c2cb24"). InnerVolumeSpecName "kube-api-access-4qpl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:10:51.600321 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:51.600285    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qpl8\" (UniqueName: \"kubernetes.io/projected/33998f16-e1aa-4e8f-adb0-e5e767c2cb24-kube-api-access-4qpl8\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:10:51.635475 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:51.635433    2587 patch_prober.go:28] interesting pod/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 11 22:10:51.635689 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:51.635501    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2" podUID="07be57ee-7747-4608-820d-b20afeaf3bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 11 22:10:51.917810 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:51.917772    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7681-4c9986"]
Sep 11 22:10:51.952825 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:51.952385    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7681-4c9986"]
Sep 11 22:10:52.232739 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:52.232498    2587 generic.go:356] "Generic (PLEG): container finished" podID="33998f16-e1aa-4e8f-adb0-e5e767c2cb24" containerID="62904f76cb6770a412bafb37c9b02b8a69b932b7c73ff921e12c570c320b0e4f" exitCode=137
Sep 11 22:10:52.233211 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:52.233043    2587 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7681-4c9986" container="test-curl-7681-4c9986"
Sep 11 22:10:52.233280 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:52.232651    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-dfa50e"
Sep 11 22:10:52.233633 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:52.232570    2587 scope.go:117] "RemoveContainer" containerID="62904f76cb6770a412bafb37c9b02b8a69b932b7c73ff921e12c570c320b0e4f"
Sep 11 22:10:52.251524 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:52.250396    2587 scope.go:117] "RemoveContainer" containerID="62904f76cb6770a412bafb37c9b02b8a69b932b7c73ff921e12c570c320b0e4f"
Sep 11 22:10:52.251524 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:52.250784    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62904f76cb6770a412bafb37c9b02b8a69b932b7c73ff921e12c570c320b0e4f\": container with ID starting with 62904f76cb6770a412bafb37c9b02b8a69b932b7c73ff921e12c570c320b0e4f not found: ID does not exist" containerID="62904f76cb6770a412bafb37c9b02b8a69b932b7c73ff921e12c570c320b0e4f"
Sep 11 22:10:52.251524 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:52.250822    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62904f76cb6770a412bafb37c9b02b8a69b932b7c73ff921e12c570c320b0e4f"} err="failed to get container status \"62904f76cb6770a412bafb37c9b02b8a69b932b7c73ff921e12c570c320b0e4f\": rpc error: code = NotFound desc = could not find container \"62904f76cb6770a412bafb37c9b02b8a69b932b7c73ff921e12c570c320b0e4f\": container with ID starting with 62904f76cb6770a412bafb37c9b02b8a69b932b7c73ff921e12c570c320b0e4f not found: ID does not exist"
Sep 11 22:10:52.253529 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:52.253495    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-dfa50e\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="33998f16-e1aa-4e8f-adb0-e5e767c2cb24" pod="opendatahub/test-curl-7681-dfa50e"
Sep 11 22:10:52.635846 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:52.635737    2587 patch_prober.go:28] interesting pod/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 11 22:10:52.635846 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:52.635810    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2" podUID="07be57ee-7747-4608-820d-b20afeaf3bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 11 22:10:52.780071 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:52.780031    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"]
Sep 11 22:10:52.780614 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:52.780587    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9b42bb2-41ad-4e9e-b41b-835186d66ed3" containerName="test-curl-7681-4c9986"
Sep 11 22:10:52.780614 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:52.780606    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9b42bb2-41ad-4e9e-b41b-835186d66ed3" containerName="test-curl-7681-4c9986"
Sep 11 22:10:52.780774 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:52.780681    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9b42bb2-41ad-4e9e-b41b-835186d66ed3" containerName="test-curl-7681-4c9986"
Sep 11 22:10:52.788641 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:52.788619    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:52.792173 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:52.791981    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-92d3298e-openshift-default-dockercfg-2tg6m\""
Sep 11 22:10:52.795431 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:52.795409    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"]
Sep 11 22:10:52.811405 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:52.811373    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-dfa50e\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="33998f16-e1aa-4e8f-adb0-e5e767c2cb24" pod="opendatahub/test-curl-7681-dfa50e"
Sep 11 22:10:52.915883 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:52.915848    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-istio-data\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:52.916053 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:52.915909    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-istio-envoy\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:52.916053 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:52.916003    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb764f30-e047-4f61-a232-9188f54b7c04-istio-podinfo\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:52.916197 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:52.916112    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k66cd\" (UniqueName: \"kubernetes.io/projected/eb764f30-e047-4f61-a232-9188f54b7c04-kube-api-access-k66cd\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:52.916197 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:52.916145    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-workload-socket\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:52.916313 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:52.916220    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb764f30-e047-4f61-a232-9188f54b7c04-istiod-ca-cert\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:52.916313 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:52.916250    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-workload-certs\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:52.916459 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:52.916319    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb764f30-e047-4f61-a232-9188f54b7c04-istio-token\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:52.916459 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:52.916393    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-credential-socket\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:53.016954 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.016913    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k66cd\" (UniqueName: \"kubernetes.io/projected/eb764f30-e047-4f61-a232-9188f54b7c04-kube-api-access-k66cd\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:53.016954 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.016953    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-workload-socket\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:53.017216 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.016989    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb764f30-e047-4f61-a232-9188f54b7c04-istiod-ca-cert\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:53.017216 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.017010    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-workload-certs\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:53.017216 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.017142    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb764f30-e047-4f61-a232-9188f54b7c04-istio-token\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:53.017216 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.017200    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-credential-socket\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:53.017512 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.017283    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-istio-data\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:53.017512 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.017403    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-istio-envoy\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:53.017512 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.017427    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-workload-socket\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:53.017672 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.017543    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb764f30-e047-4f61-a232-9188f54b7c04-istio-podinfo\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:53.017672 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.017638    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-credential-socket\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:53.017782 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.017684    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-istio-data\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:53.017974 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.017953    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-workload-certs\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:53.018134 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.018117    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb764f30-e047-4f61-a232-9188f54b7c04-istiod-ca-cert\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:53.027350 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.024722    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-istio-envoy\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:53.027350 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.024976    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb764f30-e047-4f61-a232-9188f54b7c04-istio-podinfo\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:53.033349 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.028863    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb764f30-e047-4f61-a232-9188f54b7c04-istio-token\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:53.033349 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.029518    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k66cd\" (UniqueName: \"kubernetes.io/projected/eb764f30-e047-4f61-a232-9188f54b7c04-kube-api-access-k66cd\") pod \"e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") " pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:53.103566 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.103518    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:53.242809 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.242570    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7681-4c9986" podUID="b9b42bb2-41ad-4e9e-b41b-835186d66ed3" containerName="test-curl-7681-4c9986" containerID="cri-o://4fba854fc800af89883d77f5d3ef29df9b49bcce43ccdec2ef9ff65c98fc8040" gracePeriod=2
Sep 11 22:10:53.245133 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:53.245107    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-4c9986\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="b9b42bb2-41ad-4e9e-b41b-835186d66ed3" pod="opendatahub/test-curl-7681-4c9986"
Sep 11 22:10:53.247218 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:53.247182    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-dfa50e\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="33998f16-e1aa-4e8f-adb0-e5e767c2cb24" pod="opendatahub/test-curl-7681-dfa50e"
Sep 11 22:10:53.339774 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.339376    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33998f16-e1aa-4e8f-adb0-e5e767c2cb24" path="/var/lib/kubelet/pods/33998f16-e1aa-4e8f-adb0-e5e767c2cb24/volumes"
Sep 11 22:10:53.391510 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:10:53.391483    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb764f30_e047_4f61_a232_9188f54b7c04.slice/crio-04c9f57940cf84896a83f7c887eb3f78a33c0d47017257a24c3d280d1632bde4 WatchSource:0}: Error finding container 04c9f57940cf84896a83f7c887eb3f78a33c0d47017257a24c3d280d1632bde4: Status 404 returned error can't find the container with id 04c9f57940cf84896a83f7c887eb3f78a33c0d47017257a24c3d280d1632bde4
Sep 11 22:10:53.391653 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.391586    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"]
Sep 11 22:10:53.394086 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.394051    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:10:53.394189 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.394133    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:10:53.394189 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.394180    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:10:53.429296 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.428846    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-0f3f2d"
Sep 11 22:10:53.523522 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.523429    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsgf8\" (UniqueName: \"kubernetes.io/projected/4f79a3e6-94c3-4b42-972e-ea89018c54f3-kube-api-access-gsgf8\") pod \"4f79a3e6-94c3-4b42-972e-ea89018c54f3\" (UID: \"4f79a3e6-94c3-4b42-972e-ea89018c54f3\") "
Sep 11 22:10:53.525737 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.525705    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f79a3e6-94c3-4b42-972e-ea89018c54f3-kube-api-access-gsgf8" (OuterVolumeSpecName: "kube-api-access-gsgf8") pod "4f79a3e6-94c3-4b42-972e-ea89018c54f3" (UID: "4f79a3e6-94c3-4b42-972e-ea89018c54f3"). InnerVolumeSpecName "kube-api-access-gsgf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:10:53.625100 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.625053    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsgf8\" (UniqueName: \"kubernetes.io/projected/4f79a3e6-94c3-4b42-972e-ea89018c54f3-kube-api-access-gsgf8\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:10:53.641058 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.641022    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:53.641475 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.641443    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:53.642299 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:53.642275    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:10:54.249533 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:54.249501    2587 generic.go:356] "Generic (PLEG): container finished" podID="4f79a3e6-94c3-4b42-972e-ea89018c54f3" containerID="7eaef6f2a98dec4f49727ce37a9030f310d4a7395af47e0f3936a43d2bf68779" exitCode=137
Sep 11 22:10:54.249969 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:54.249545    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-0f3f2d"
Sep 11 22:10:54.249969 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:54.249601    2587 scope.go:117] "RemoveContainer" containerID="7eaef6f2a98dec4f49727ce37a9030f310d4a7395af47e0f3936a43d2bf68779"
Sep 11 22:10:54.251342 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:54.251279    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn" event={"ID":"eb764f30-e047-4f61-a232-9188f54b7c04","Type":"ContainerStarted","Data":"3128b007868ff5cf480df4ec024286ad26c700a81a7619f22484ef7cb064a841"}
Sep 11 22:10:54.251456 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:54.251380    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn" event={"ID":"eb764f30-e047-4f61-a232-9188f54b7c04","Type":"ContainerStarted","Data":"04c9f57940cf84896a83f7c887eb3f78a33c0d47017257a24c3d280d1632bde4"}
Sep 11 22:10:54.259148 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:54.259130    2587 scope.go:117] "RemoveContainer" containerID="7eaef6f2a98dec4f49727ce37a9030f310d4a7395af47e0f3936a43d2bf68779"
Sep 11 22:10:54.259437 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:54.259418    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eaef6f2a98dec4f49727ce37a9030f310d4a7395af47e0f3936a43d2bf68779\": container with ID starting with 7eaef6f2a98dec4f49727ce37a9030f310d4a7395af47e0f3936a43d2bf68779 not found: ID does not exist" containerID="7eaef6f2a98dec4f49727ce37a9030f310d4a7395af47e0f3936a43d2bf68779"
Sep 11 22:10:54.259508 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:54.259449    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eaef6f2a98dec4f49727ce37a9030f310d4a7395af47e0f3936a43d2bf68779"} err="failed to get container status \"7eaef6f2a98dec4f49727ce37a9030f310d4a7395af47e0f3936a43d2bf68779\": rpc error: code = NotFound desc = could not find container \"7eaef6f2a98dec4f49727ce37a9030f310d4a7395af47e0f3936a43d2bf68779\": container with ID starting with 7eaef6f2a98dec4f49727ce37a9030f310d4a7395af47e0f3936a43d2bf68779 not found: ID does not exist"
Sep 11 22:10:54.273098 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:54.273070    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-0f3f2d\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="4f79a3e6-94c3-4b42-972e-ea89018c54f3" pod="opendatahub/test-curl-7681-0f3f2d"
Sep 11 22:10:54.273435 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:54.273398    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn" podStartSLOduration=2.273385761 podStartE2EDuration="2.273385761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:10:54.270994418 +0000 UTC m=+1107.446070803" watchObservedRunningTime="2026-09-11 22:10:54.273385761 +0000 UTC m=+1107.448462130"
Sep 11 22:10:55.104117 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:55.104079    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:55.109372 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:55.109347    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:55.131956 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:55.131913    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-0f3f2d\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="4f79a3e6-94c3-4b42-972e-ea89018c54f3" pod="opendatahub/test-curl-7681-0f3f2d"
Sep 11 22:10:55.258586 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:55.258551    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:55.259651 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:55.259634    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:10:55.261999 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:55.261972    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-0f3f2d\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="4f79a3e6-94c3-4b42-972e-ea89018c54f3" pod="opendatahub/test-curl-7681-0f3f2d"
Sep 11 22:10:55.338657 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:55.338623    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f79a3e6-94c3-4b42-972e-ea89018c54f3" path="/var/lib/kubelet/pods/4f79a3e6-94c3-4b42-972e-ea89018c54f3/volumes"
Sep 11 22:10:55.413915 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:55.413887    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-4c9986"
Sep 11 22:10:55.545880 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:55.545842    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4q9g\" (UniqueName: \"kubernetes.io/projected/b9b42bb2-41ad-4e9e-b41b-835186d66ed3-kube-api-access-l4q9g\") pod \"b9b42bb2-41ad-4e9e-b41b-835186d66ed3\" (UID: \"b9b42bb2-41ad-4e9e-b41b-835186d66ed3\") "
Sep 11 22:10:55.547964 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:55.547932    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b42bb2-41ad-4e9e-b41b-835186d66ed3-kube-api-access-l4q9g" (OuterVolumeSpecName: "kube-api-access-l4q9g") pod "b9b42bb2-41ad-4e9e-b41b-835186d66ed3" (UID: "b9b42bb2-41ad-4e9e-b41b-835186d66ed3"). InnerVolumeSpecName "kube-api-access-l4q9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:10:55.646975 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:55.646898    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4q9g\" (UniqueName: \"kubernetes.io/projected/b9b42bb2-41ad-4e9e-b41b-835186d66ed3-kube-api-access-l4q9g\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:10:56.271389 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.271349    2587 generic.go:356] "Generic (PLEG): container finished" podID="b9b42bb2-41ad-4e9e-b41b-835186d66ed3" containerID="4fba854fc800af89883d77f5d3ef29df9b49bcce43ccdec2ef9ff65c98fc8040" exitCode=137
Sep 11 22:10:56.272109 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.272048    2587 scope.go:117] "RemoveContainer" containerID="4fba854fc800af89883d77f5d3ef29df9b49bcce43ccdec2ef9ff65c98fc8040"
Sep 11 22:10:56.272109 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.272110    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7681-4c9986"
Sep 11 22:10:56.283567 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.283538    2587 scope.go:117] "RemoveContainer" containerID="4fba854fc800af89883d77f5d3ef29df9b49bcce43ccdec2ef9ff65c98fc8040"
Sep 11 22:10:56.283842 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:56.283821    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fba854fc800af89883d77f5d3ef29df9b49bcce43ccdec2ef9ff65c98fc8040\": container with ID starting with 4fba854fc800af89883d77f5d3ef29df9b49bcce43ccdec2ef9ff65c98fc8040 not found: ID does not exist" containerID="4fba854fc800af89883d77f5d3ef29df9b49bcce43ccdec2ef9ff65c98fc8040"
Sep 11 22:10:56.283923 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.283852    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fba854fc800af89883d77f5d3ef29df9b49bcce43ccdec2ef9ff65c98fc8040"} err="failed to get container status \"4fba854fc800af89883d77f5d3ef29df9b49bcce43ccdec2ef9ff65c98fc8040\": rpc error: code = NotFound desc = could not find container \"4fba854fc800af89883d77f5d3ef29df9b49bcce43ccdec2ef9ff65c98fc8040\": container with ID starting with 4fba854fc800af89883d77f5d3ef29df9b49bcce43ccdec2ef9ff65c98fc8040 not found: ID does not exist"
Sep 11 22:10:56.285132 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:56.285109    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-4c9986\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="b9b42bb2-41ad-4e9e-b41b-835186d66ed3" pod="opendatahub/test-curl-7681-4c9986"
Sep 11 22:10:56.543907 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.543879    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:10:56.570858 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:56.570809    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-4c9986\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="b9b42bb2-41ad-4e9e-b41b-835186d66ed3" pod="opendatahub/test-curl-7681-4c9986"
Sep 11 22:10:56.657243 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.657212    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istio-data\") pod \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") "
Sep 11 22:10:56.657440 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.657272    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2w4c\" (UniqueName: \"kubernetes.io/projected/f2f9ee75-e2ab-45e6-9f74-9088f1508251-kube-api-access-q2w4c\") pod \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") "
Sep 11 22:10:56.657440 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.657305    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-workload-socket\") pod \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") "
Sep 11 22:10:56.657440 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.657379    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istio-token\") pod \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") "
Sep 11 22:10:56.657440 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.657403    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istio-envoy\") pod \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") "
Sep 11 22:10:56.657636 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.657453    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istio-podinfo\") pod \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") "
Sep 11 22:10:56.657636 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.657508    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istiod-ca-cert\") pod \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") "
Sep 11 22:10:56.657636 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.657548    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istio-data" (OuterVolumeSpecName: "istio-data") pod "f2f9ee75-e2ab-45e6-9f74-9088f1508251" (UID: "f2f9ee75-e2ab-45e6-9f74-9088f1508251"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:10:56.657636 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.657610    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-workload-certs\") pod \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") "
Sep 11 22:10:56.657636 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.657637    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-credential-socket\") pod \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\" (UID: \"f2f9ee75-e2ab-45e6-9f74-9088f1508251\") "
Sep 11 22:10:56.657895 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.657643    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f2f9ee75-e2ab-45e6-9f74-9088f1508251" (UID: "f2f9ee75-e2ab-45e6-9f74-9088f1508251"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:10:56.657895 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.657881    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f2f9ee75-e2ab-45e6-9f74-9088f1508251" (UID: "f2f9ee75-e2ab-45e6-9f74-9088f1508251"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:10:56.658016 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.657897    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f2f9ee75-e2ab-45e6-9f74-9088f1508251" (UID: "f2f9ee75-e2ab-45e6-9f74-9088f1508251"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:10:56.658073 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.658058    2587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istiod-ca-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:10:56.658128 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.658077    2587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-workload-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:10:56.658128 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.658089    2587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istio-data\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:10:56.658128 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.658100    2587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-workload-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:10:56.658128 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.658057    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f2f9ee75-e2ab-45e6-9f74-9088f1508251" (UID: "f2f9ee75-e2ab-45e6-9f74-9088f1508251"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:10:56.659890 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.659863    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f2f9ee75-e2ab-45e6-9f74-9088f1508251" (UID: "f2f9ee75-e2ab-45e6-9f74-9088f1508251"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:10:56.660027 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.659880    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istio-token" (OuterVolumeSpecName: "istio-token") pod "f2f9ee75-e2ab-45e6-9f74-9088f1508251" (UID: "f2f9ee75-e2ab-45e6-9f74-9088f1508251"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:10:56.660027 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.659978    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f2f9ee75-e2ab-45e6-9f74-9088f1508251" (UID: "f2f9ee75-e2ab-45e6-9f74-9088f1508251"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:10:56.660104 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.660067    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f9ee75-e2ab-45e6-9f74-9088f1508251-kube-api-access-q2w4c" (OuterVolumeSpecName: "kube-api-access-q2w4c") pod "f2f9ee75-e2ab-45e6-9f74-9088f1508251" (UID: "f2f9ee75-e2ab-45e6-9f74-9088f1508251"). InnerVolumeSpecName "kube-api-access-q2w4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:10:56.759448 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.759413    2587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istio-token\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:10:56.759448 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.759442    2587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istio-envoy\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:10:56.759448 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.759451    2587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2f9ee75-e2ab-45e6-9f74-9088f1508251-istio-podinfo\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:10:56.759448 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.759459    2587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2f9ee75-e2ab-45e6-9f74-9088f1508251-credential-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:10:56.759730 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:56.759469    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2w4c\" (UniqueName: \"kubernetes.io/projected/f2f9ee75-e2ab-45e6-9f74-9088f1508251-kube-api-access-q2w4c\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:10:57.278962 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:57.278924    2587 generic.go:356] "Generic (PLEG): container finished" podID="f2f9ee75-e2ab-45e6-9f74-9088f1508251" containerID="1063e304cdbd52beb4e3c869d51be44ceedf903bea53f7e83f14d679a293e932" exitCode=0
Sep 11 22:10:57.279452 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:57.278996    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"
Sep 11 22:10:57.279452 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:57.279045    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq" event={"ID":"f2f9ee75-e2ab-45e6-9f74-9088f1508251","Type":"ContainerDied","Data":"1063e304cdbd52beb4e3c869d51be44ceedf903bea53f7e83f14d679a293e932"}
Sep 11 22:10:57.279452 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:57.279220    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq" event={"ID":"f2f9ee75-e2ab-45e6-9f74-9088f1508251","Type":"ContainerDied","Data":"c570cc0b3880e2bdce62e260dee3814b8568991c32ea8d073136d31e0021a7a7"}
Sep 11 22:10:57.279452 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:57.279259    2587 scope.go:117] "RemoveContainer" containerID="1063e304cdbd52beb4e3c869d51be44ceedf903bea53f7e83f14d679a293e932"
Sep 11 22:10:57.288079 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:57.288042    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-4c9986\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="b9b42bb2-41ad-4e9e-b41b-835186d66ed3" pod="opendatahub/test-curl-7681-4c9986"
Sep 11 22:10:57.295690 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:57.295668    2587 scope.go:117] "RemoveContainer" containerID="d8c1f7b71fe6572df3397251cfd4a567d0ec3d9fabd779d5cae1031ff5b00699"
Sep 11 22:10:57.304025 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:57.304004    2587 scope.go:117] "RemoveContainer" containerID="1063e304cdbd52beb4e3c869d51be44ceedf903bea53f7e83f14d679a293e932"
Sep 11 22:10:57.304307 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:57.304285    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1063e304cdbd52beb4e3c869d51be44ceedf903bea53f7e83f14d679a293e932\": container with ID starting with 1063e304cdbd52beb4e3c869d51be44ceedf903bea53f7e83f14d679a293e932 not found: ID does not exist" containerID="1063e304cdbd52beb4e3c869d51be44ceedf903bea53f7e83f14d679a293e932"
Sep 11 22:10:57.304390 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:57.304319    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1063e304cdbd52beb4e3c869d51be44ceedf903bea53f7e83f14d679a293e932"} err="failed to get container status \"1063e304cdbd52beb4e3c869d51be44ceedf903bea53f7e83f14d679a293e932\": rpc error: code = NotFound desc = could not find container \"1063e304cdbd52beb4e3c869d51be44ceedf903bea53f7e83f14d679a293e932\": container with ID starting with 1063e304cdbd52beb4e3c869d51be44ceedf903bea53f7e83f14d679a293e932 not found: ID does not exist"
Sep 11 22:10:57.304390 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:57.304355    2587 scope.go:117] "RemoveContainer" containerID="d8c1f7b71fe6572df3397251cfd4a567d0ec3d9fabd779d5cae1031ff5b00699"
Sep 11 22:10:57.304641 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:57.304621    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8c1f7b71fe6572df3397251cfd4a567d0ec3d9fabd779d5cae1031ff5b00699\": container with ID starting with d8c1f7b71fe6572df3397251cfd4a567d0ec3d9fabd779d5cae1031ff5b00699 not found: ID does not exist" containerID="d8c1f7b71fe6572df3397251cfd4a567d0ec3d9fabd779d5cae1031ff5b00699"
Sep 11 22:10:57.304707 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:57.304645    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c1f7b71fe6572df3397251cfd4a567d0ec3d9fabd779d5cae1031ff5b00699"} err="failed to get container status \"d8c1f7b71fe6572df3397251cfd4a567d0ec3d9fabd779d5cae1031ff5b00699\": rpc error: code = NotFound desc = could not find container \"d8c1f7b71fe6572df3397251cfd4a567d0ec3d9fabd779d5cae1031ff5b00699\": container with ID starting with d8c1f7b71fe6572df3397251cfd4a567d0ec3d9fabd779d5cae1031ff5b00699 not found: ID does not exist"
Sep 11 22:10:57.307885 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:57.307865    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"]
Sep 11 22:10:57.311730 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:57.311706    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-4c9986\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="b9b42bb2-41ad-4e9e-b41b-835186d66ed3" pod="opendatahub/test-curl-7681-4c9986"
Sep 11 22:10:57.312005 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:57.311986    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-27dad9-openshift-default-5f5c684bbd-zjmcq"]
Sep 11 22:10:57.334036 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:10:57.334004    2587 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7681-4c9986\" is forbidden: User \"system:node:ip-10-0-139-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-88.ec2.internal' and this object" podUID="b9b42bb2-41ad-4e9e-b41b-835186d66ed3" pod="opendatahub/test-curl-7681-4c9986"
Sep 11 22:10:57.337242 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:57.337215    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9b42bb2-41ad-4e9e-b41b-835186d66ed3" path="/var/lib/kubelet/pods/b9b42bb2-41ad-4e9e-b41b-835186d66ed3/volumes"
Sep 11 22:10:57.337734 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:10:57.337716    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2f9ee75-e2ab-45e6-9f74-9088f1508251" path="/var/lib/kubelet/pods/f2f9ee75-e2ab-45e6-9f74-9088f1508251/volumes"
Sep 11 22:11:01.694117 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:01.694082    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh"]
Sep 11 22:11:01.694613 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:01.694590    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2f9ee75-e2ab-45e6-9f74-9088f1508251" containerName="istio-proxy"
Sep 11 22:11:01.694613 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:01.694605    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2f9ee75-e2ab-45e6-9f74-9088f1508251" containerName="istio-proxy"
Sep 11 22:11:01.694751 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:01.694617    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2f9ee75-e2ab-45e6-9f74-9088f1508251" containerName="istio-proxy"
Sep 11 22:11:01.694751 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:01.694624    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2f9ee75-e2ab-45e6-9f74-9088f1508251" containerName="istio-proxy"
Sep 11 22:11:01.694751 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:01.694704    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2f9ee75-e2ab-45e6-9f74-9088f1508251" containerName="istio-proxy"
Sep 11 22:11:01.699674 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:01.699651    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh"
Sep 11 22:11:01.702550 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:01.702526    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-92d3298e-serving-cert\""
Sep 11 22:11:01.712845 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:01.712813    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh"]
Sep 11 22:11:01.811921 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:01.811837    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e32e4ca9-7442-4025-974c-77c9a57878d5-openshift-service-ca\") pod \"maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh\" (UID: \"e32e4ca9-7442-4025-974c-77c9a57878d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh"
Sep 11 22:11:01.811921 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:01.811902    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e32e4ca9-7442-4025-974c-77c9a57878d5-metrics-certs\") pod \"maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh\" (UID: \"e32e4ca9-7442-4025-974c-77c9a57878d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh"
Sep 11 22:11:01.811921 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:01.811932    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd5km\" (UniqueName: \"kubernetes.io/projected/e32e4ca9-7442-4025-974c-77c9a57878d5-kube-api-access-bd5km\") pod \"maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh\" (UID: \"e32e4ca9-7442-4025-974c-77c9a57878d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh"
Sep 11 22:11:01.812309 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:01.811971    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e32e4ca9-7442-4025-974c-77c9a57878d5-maas-api-tls\") pod \"maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh\" (UID: \"e32e4ca9-7442-4025-974c-77c9a57878d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh"
Sep 11 22:11:01.913459 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:01.913427    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e32e4ca9-7442-4025-974c-77c9a57878d5-maas-api-tls\") pod \"maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh\" (UID: \"e32e4ca9-7442-4025-974c-77c9a57878d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh"
Sep 11 22:11:01.913624 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:01.913560    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e32e4ca9-7442-4025-974c-77c9a57878d5-openshift-service-ca\") pod \"maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh\" (UID: \"e32e4ca9-7442-4025-974c-77c9a57878d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh"
Sep 11 22:11:01.913624 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:11:01.913579    2587 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-serving-cert: secret "maas-api-e2e-mt-92d3298e-serving-cert" not found
Sep 11 22:11:01.913624 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:01.913589    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e32e4ca9-7442-4025-974c-77c9a57878d5-metrics-certs\") pod \"maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh\" (UID: \"e32e4ca9-7442-4025-974c-77c9a57878d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh"
Sep 11 22:11:01.913624 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:01.913605    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd5km\" (UniqueName: \"kubernetes.io/projected/e32e4ca9-7442-4025-974c-77c9a57878d5-kube-api-access-bd5km\") pod \"maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh\" (UID: \"e32e4ca9-7442-4025-974c-77c9a57878d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh"
Sep 11 22:11:01.913819 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:11:01.913640    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e32e4ca9-7442-4025-974c-77c9a57878d5-maas-api-tls podName:e32e4ca9-7442-4025-974c-77c9a57878d5 nodeName:}" failed. No retries permitted until 2026-09-11 22:11:02.413624105 +0000 UTC m=+1115.588700454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e32e4ca9-7442-4025-974c-77c9a57878d5-maas-api-tls") pod "maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh" (UID: "e32e4ca9-7442-4025-974c-77c9a57878d5") : secret "maas-api-e2e-mt-92d3298e-serving-cert" not found
Sep 11 22:11:01.914395 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:01.914359    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e32e4ca9-7442-4025-974c-77c9a57878d5-openshift-service-ca\") pod \"maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh\" (UID: \"e32e4ca9-7442-4025-974c-77c9a57878d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh"
Sep 11 22:11:01.916180 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:01.916153    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e32e4ca9-7442-4025-974c-77c9a57878d5-metrics-certs\") pod \"maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh\" (UID: \"e32e4ca9-7442-4025-974c-77c9a57878d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh"
Sep 11 22:11:01.924555 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:01.924517    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd5km\" (UniqueName: \"kubernetes.io/projected/e32e4ca9-7442-4025-974c-77c9a57878d5-kube-api-access-bd5km\") pod \"maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh\" (UID: \"e32e4ca9-7442-4025-974c-77c9a57878d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh"
Sep 11 22:11:02.420913 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:02.420863    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e32e4ca9-7442-4025-974c-77c9a57878d5-maas-api-tls\") pod \"maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh\" (UID: \"e32e4ca9-7442-4025-974c-77c9a57878d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh"
Sep 11 22:11:02.424250 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:02.424220    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e32e4ca9-7442-4025-974c-77c9a57878d5-maas-api-tls\") pod \"maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh\" (UID: \"e32e4ca9-7442-4025-974c-77c9a57878d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh"
Sep 11 22:11:02.472463 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:02.472426    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"]
Sep 11 22:11:02.472722 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:02.472673    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j" podUID="9babb15f-7478-4134-aefc-9fefac8afd03" containerName="maas-api" containerID="cri-o://88981b5c9f5c397b51d0593221ec684d76285e7c61e706c355030152cce646b5" gracePeriod=30
Sep 11 22:11:02.613551 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:02.613508    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh"
Sep 11 22:11:02.748232 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:02.748204    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"
Sep 11 22:11:02.825072 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:02.825027    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9babb15f-7478-4134-aefc-9fefac8afd03-maas-api-tls\") pod \"9babb15f-7478-4134-aefc-9fefac8afd03\" (UID: \"9babb15f-7478-4134-aefc-9fefac8afd03\") "
Sep 11 22:11:02.825273 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:02.825209    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9babb15f-7478-4134-aefc-9fefac8afd03-metrics-certs\") pod \"9babb15f-7478-4134-aefc-9fefac8afd03\" (UID: \"9babb15f-7478-4134-aefc-9fefac8afd03\") "
Sep 11 22:11:02.825373 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:02.825278    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9babb15f-7478-4134-aefc-9fefac8afd03-openshift-service-ca\") pod \"9babb15f-7478-4134-aefc-9fefac8afd03\" (UID: \"9babb15f-7478-4134-aefc-9fefac8afd03\") "
Sep 11 22:11:02.825373 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:02.825311    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzf2g\" (UniqueName: \"kubernetes.io/projected/9babb15f-7478-4134-aefc-9fefac8afd03-kube-api-access-hzf2g\") pod \"9babb15f-7478-4134-aefc-9fefac8afd03\" (UID: \"9babb15f-7478-4134-aefc-9fefac8afd03\") "
Sep 11 22:11:02.825792 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:02.825687    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9babb15f-7478-4134-aefc-9fefac8afd03-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9babb15f-7478-4134-aefc-9fefac8afd03" (UID: "9babb15f-7478-4134-aefc-9fefac8afd03"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:11:02.827759 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:02.827724    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9babb15f-7478-4134-aefc-9fefac8afd03-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9babb15f-7478-4134-aefc-9fefac8afd03" (UID: "9babb15f-7478-4134-aefc-9fefac8afd03"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:11:02.828106 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:02.828068    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9babb15f-7478-4134-aefc-9fefac8afd03-kube-api-access-hzf2g" (OuterVolumeSpecName: "kube-api-access-hzf2g") pod "9babb15f-7478-4134-aefc-9fefac8afd03" (UID: "9babb15f-7478-4134-aefc-9fefac8afd03"). InnerVolumeSpecName "kube-api-access-hzf2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:11:02.828106 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:02.828079    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9babb15f-7478-4134-aefc-9fefac8afd03-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9babb15f-7478-4134-aefc-9fefac8afd03" (UID: "9babb15f-7478-4134-aefc-9fefac8afd03"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:11:02.902003 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:02.901973    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh"]
Sep 11 22:11:02.903493 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:11:02.903469    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32e4ca9_7442_4025_974c_77c9a57878d5.slice/crio-f33690a891246e9149a10a49c634956f187e5f110da020d38f28dbfd3ed1a8e2 WatchSource:0}: Error finding container f33690a891246e9149a10a49c634956f187e5f110da020d38f28dbfd3ed1a8e2: Status 404 returned error can't find the container with id f33690a891246e9149a10a49c634956f187e5f110da020d38f28dbfd3ed1a8e2
Sep 11 22:11:02.927467 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:02.927444    2587 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9babb15f-7478-4134-aefc-9fefac8afd03-openshift-service-ca\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:11:02.927538 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:02.927474    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzf2g\" (UniqueName: \"kubernetes.io/projected/9babb15f-7478-4134-aefc-9fefac8afd03-kube-api-access-hzf2g\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:11:02.927538 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:02.927491    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9babb15f-7478-4134-aefc-9fefac8afd03-maas-api-tls\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:11:02.927538 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:02.927504    2587 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9babb15f-7478-4134-aefc-9fefac8afd03-metrics-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:11:03.323453 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:03.323356    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh" event={"ID":"e32e4ca9-7442-4025-974c-77c9a57878d5","Type":"ContainerStarted","Data":"f33690a891246e9149a10a49c634956f187e5f110da020d38f28dbfd3ed1a8e2"}
Sep 11 22:11:03.324912 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:03.324884    2587 generic.go:356] "Generic (PLEG): container finished" podID="9babb15f-7478-4134-aefc-9fefac8afd03" containerID="88981b5c9f5c397b51d0593221ec684d76285e7c61e706c355030152cce646b5" exitCode=0
Sep 11 22:11:03.325062 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:03.324920    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j" event={"ID":"9babb15f-7478-4134-aefc-9fefac8afd03","Type":"ContainerDied","Data":"88981b5c9f5c397b51d0593221ec684d76285e7c61e706c355030152cce646b5"}
Sep 11 22:11:03.325062 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:03.324934    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"
Sep 11 22:11:03.325062 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:03.324952    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j" event={"ID":"9babb15f-7478-4134-aefc-9fefac8afd03","Type":"ContainerDied","Data":"279ec09cebfb6ce3b666e7157b62027833a02368348c61df15adf037f6bcb38e"}
Sep 11 22:11:03.325062 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:03.324975    2587 scope.go:117] "RemoveContainer" containerID="88981b5c9f5c397b51d0593221ec684d76285e7c61e706c355030152cce646b5"
Sep 11 22:11:03.336056 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:03.336036    2587 scope.go:117] "RemoveContainer" containerID="88981b5c9f5c397b51d0593221ec684d76285e7c61e706c355030152cce646b5"
Sep 11 22:11:03.336349 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:11:03.336308    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88981b5c9f5c397b51d0593221ec684d76285e7c61e706c355030152cce646b5\": container with ID starting with 88981b5c9f5c397b51d0593221ec684d76285e7c61e706c355030152cce646b5 not found: ID does not exist" containerID="88981b5c9f5c397b51d0593221ec684d76285e7c61e706c355030152cce646b5"
Sep 11 22:11:03.336427 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:03.336350    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88981b5c9f5c397b51d0593221ec684d76285e7c61e706c355030152cce646b5"} err="failed to get container status \"88981b5c9f5c397b51d0593221ec684d76285e7c61e706c355030152cce646b5\": rpc error: code = NotFound desc = could not find container \"88981b5c9f5c397b51d0593221ec684d76285e7c61e706c355030152cce646b5\": container with ID starting with 88981b5c9f5c397b51d0593221ec684d76285e7c61e706c355030152cce646b5 not found: ID does not exist"
Sep 11 22:11:03.350210 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:03.350184    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"]
Sep 11 22:11:03.354363 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:03.354314    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-37182a-757999cc59-6jh8j"]
Sep 11 22:11:04.332741 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:04.332707    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh" event={"ID":"e32e4ca9-7442-4025-974c-77c9a57878d5","Type":"ContainerStarted","Data":"c3c28977bee453e315187b1cd74eda2399c7c4685c2ce96ba36080ab527dc4b2"}
Sep 11 22:11:04.333154 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:04.332826    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh"
Sep 11 22:11:04.357615 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:04.357558    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh" podStartSLOduration=2.770461158 podStartE2EDuration="3.357539889s" totalImagesPullingTime="587.078731ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:11:04.35617281 +0000 UTC m=+1117.531249193" watchObservedRunningTime="2026-09-11 22:11:04.357539889 +0000 UTC m=+1117.532616259"
Sep 11 22:11:05.337293 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:05.337254    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9babb15f-7478-4134-aefc-9fefac8afd03" path="/var/lib/kubelet/pods/9babb15f-7478-4134-aefc-9fefac8afd03/volumes"
Sep 11 22:11:10.344154 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:10.344117    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh"
Sep 11 22:11:15.878706 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:15.878671    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:11:25.691799 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:25.690763    2587 patch_prober.go:28] interesting pod/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": read tcp 10.134.0.2:49908->10.134.0.67:15021: read: connection reset by peer" start-of-body=
Sep 11 22:11:25.691799 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:25.690849    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn" podUID="eb764f30-e047-4f61-a232-9188f54b7c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": read tcp 10.134.0.2:49908->10.134.0.67:15021: read: connection reset by peer"
Sep 11 22:11:26.477605 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:26.477570    2587 generic.go:356] "Generic (PLEG): container finished" podID="eb764f30-e047-4f61-a232-9188f54b7c04" containerID="3128b007868ff5cf480df4ec024286ad26c700a81a7619f22484ef7cb064a841" exitCode=0
Sep 11 22:11:26.477788 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:26.477642    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn" event={"ID":"eb764f30-e047-4f61-a232-9188f54b7c04","Type":"ContainerDied","Data":"3128b007868ff5cf480df4ec024286ad26c700a81a7619f22484ef7cb064a841"}
Sep 11 22:11:26.478064 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:26.478050    2587 scope.go:117] "RemoveContainer" containerID="3128b007868ff5cf480df4ec024286ad26c700a81a7619f22484ef7cb064a841"
Sep 11 22:11:26.479182 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:26.479157    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:11:26.479236 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:26.479228    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:11:26.479271 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:26.479259    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:11:27.486165 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:27.486139    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn" event={"ID":"eb764f30-e047-4f61-a232-9188f54b7c04","Type":"ContainerStarted","Data":"1eae16b6e4f577c60e20e6efcaa6a5845e766e34fb9866f288211e1af3d43862"}
Sep 11 22:11:28.104272 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:28.104238    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:11:28.104475 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:28.104453    2587 patch_prober.go:28] interesting pod/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 11 22:11:28.104550 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:28.104531    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn" podUID="eb764f30-e047-4f61-a232-9188f54b7c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 11 22:11:29.104778 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:29.104742    2587 patch_prober.go:28] interesting pod/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 11 22:11:29.105171 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:29.104810    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn" podUID="eb764f30-e047-4f61-a232-9188f54b7c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 11 22:11:30.104684 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:30.104647    2587 patch_prober.go:28] interesting pod/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 11 22:11:30.104856 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:30.104703    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn" podUID="eb764f30-e047-4f61-a232-9188f54b7c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 11 22:11:31.104699 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:31.104662    2587 patch_prober.go:28] interesting pod/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 11 22:11:31.104867 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:31.104722    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn" podUID="eb764f30-e047-4f61-a232-9188f54b7c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 11 22:11:32.104481 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:32.104442    2587 patch_prober.go:28] interesting pod/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 11 22:11:32.104659 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:32.104514    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn" podUID="eb764f30-e047-4f61-a232-9188f54b7c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 11 22:11:33.107505 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:33.107470    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:11:33.107901 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:33.107834    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:11:33.108514 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:33.108496    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:11:45.837904 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:45.837860    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"]
Sep 11 22:11:45.839927 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:45.839897    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9babb15f-7478-4134-aefc-9fefac8afd03" containerName="maas-api"
Sep 11 22:11:45.840115 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:45.840099    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="9babb15f-7478-4134-aefc-9fefac8afd03" containerName="maas-api"
Sep 11 22:11:45.841615 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:45.841592    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2f9ee75-e2ab-45e6-9f74-9088f1508251" containerName="istio-proxy"
Sep 11 22:11:45.841754 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:45.841740    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="9babb15f-7478-4134-aefc-9fefac8afd03" containerName="maas-api"
Sep 11 22:11:45.855364 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:45.855310    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"]
Sep 11 22:11:45.855509 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:45.855475    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:45.860546 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:45.860521    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-d09c96-openshift-default-dockercfg-cl4vw\""
Sep 11 22:11:45.975243 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:45.974666    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-istio-data\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:45.975243 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:45.974721    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpb29\" (UniqueName: \"kubernetes.io/projected/64f7d428-457a-4b96-bb21-266305d3cab7-kube-api-access-mpb29\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:45.975243 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:45.974800    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64f7d428-457a-4b96-bb21-266305d3cab7-istio-podinfo\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:45.975243 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:45.974894    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-credential-socket\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:45.975243 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:45.974920    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64f7d428-457a-4b96-bb21-266305d3cab7-istiod-ca-cert\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:45.975243 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:45.974953    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-istio-envoy\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:45.975243 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:45.975004    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-workload-certs\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:45.975243 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:45.975043    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-workload-socket\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:45.975243 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:45.975078    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64f7d428-457a-4b96-bb21-266305d3cab7-istio-token\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:46.076139 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.076102    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-credential-socket\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:46.076139 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.076138    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64f7d428-457a-4b96-bb21-266305d3cab7-istiod-ca-cert\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:46.076401 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.076168    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-istio-envoy\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:46.076401 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.076214    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-workload-certs\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:46.076401 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.076253    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-workload-socket\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:46.076401 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.076283    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64f7d428-457a-4b96-bb21-266305d3cab7-istio-token\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:46.076711 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.076674    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-workload-certs\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:46.076711 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.076707    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-workload-socket\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:46.076904 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.076673    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-credential-socket\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:46.076904 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.076741    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-istio-data\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:46.076904 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.076770    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpb29\" (UniqueName: \"kubernetes.io/projected/64f7d428-457a-4b96-bb21-266305d3cab7-kube-api-access-mpb29\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:46.076904 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.076856    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64f7d428-457a-4b96-bb21-266305d3cab7-istio-podinfo\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:46.077134 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.076991    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-istio-data\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:46.077269 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.077244    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64f7d428-457a-4b96-bb21-266305d3cab7-istiod-ca-cert\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:46.079463 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.079437    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-istio-envoy\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:46.079670 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.079651    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64f7d428-457a-4b96-bb21-266305d3cab7-istio-podinfo\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:46.084637 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.084612    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64f7d428-457a-4b96-bb21-266305d3cab7-istio-token\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:46.084738 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.084721    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpb29\" (UniqueName: \"kubernetes.io/projected/64f7d428-457a-4b96-bb21-266305d3cab7-kube-api-access-mpb29\") pod \"e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") " pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:46.173777 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.173735    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:46.453982 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.453951    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"]
Sep 11 22:11:46.455771 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:11:46.455743    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f7d428_457a_4b96_bb21_266305d3cab7.slice/crio-7c6c3609a018637c702ca52d07f82fbd058199fa2f5be96a9587598c0c446f6d WatchSource:0}: Error finding container 7c6c3609a018637c702ca52d07f82fbd058199fa2f5be96a9587598c0c446f6d: Status 404 returned error can't find the container with id 7c6c3609a018637c702ca52d07f82fbd058199fa2f5be96a9587598c0c446f6d
Sep 11 22:11:46.458445 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.458412    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:11:46.458539 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.458479    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:11:46.458539 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.458506    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:11:46.592337 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.592286    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j" event={"ID":"64f7d428-457a-4b96-bb21-266305d3cab7","Type":"ContainerStarted","Data":"a604650a1a4e11f913267cc4f25d3acee64b5313d8912e393d2d9ba8110c0409"}
Sep 11 22:11:46.592527 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.592359    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j" event={"ID":"64f7d428-457a-4b96-bb21-266305d3cab7","Type":"ContainerStarted","Data":"7c6c3609a018637c702ca52d07f82fbd058199fa2f5be96a9587598c0c446f6d"}
Sep 11 22:11:46.613623 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:46.613577    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j" podStartSLOduration=1.613560063 podStartE2EDuration="1.613560063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:11:46.611409208 +0000 UTC m=+1159.786485600" watchObservedRunningTime="2026-09-11 22:11:46.613560063 +0000 UTC m=+1159.788636433"
Sep 11 22:11:47.174127 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:47.174094    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:47.179667 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:47.179644    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:47.600106 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:47.600014    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:47.604402 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:47.604379    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:11:52.482117 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:52.482032    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5"]
Sep 11 22:11:52.486861 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:52.486834    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5"
Sep 11 22:11:52.489553 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:52.489526    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-d09c96-serving-cert\""
Sep 11 22:11:52.495753 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:52.495724    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5"]
Sep 11 22:11:52.543103 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:52.543063    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06f1362b-5ccc-4ca7-8239-e5982339df97-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5\" (UID: \"06f1362b-5ccc-4ca7-8239-e5982339df97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5"
Sep 11 22:11:52.543310 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:52.543176    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06f1362b-5ccc-4ca7-8239-e5982339df97-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5\" (UID: \"06f1362b-5ccc-4ca7-8239-e5982339df97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5"
Sep 11 22:11:52.543310 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:52.543262    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06f1362b-5ccc-4ca7-8239-e5982339df97-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5\" (UID: \"06f1362b-5ccc-4ca7-8239-e5982339df97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5"
Sep 11 22:11:52.543310 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:52.543300    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnk6k\" (UniqueName: \"kubernetes.io/projected/06f1362b-5ccc-4ca7-8239-e5982339df97-kube-api-access-nnk6k\") pod \"maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5\" (UID: \"06f1362b-5ccc-4ca7-8239-e5982339df97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5"
Sep 11 22:11:52.644846 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:52.644803    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06f1362b-5ccc-4ca7-8239-e5982339df97-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5\" (UID: \"06f1362b-5ccc-4ca7-8239-e5982339df97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5"
Sep 11 22:11:52.645072 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:52.644866    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06f1362b-5ccc-4ca7-8239-e5982339df97-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5\" (UID: \"06f1362b-5ccc-4ca7-8239-e5982339df97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5"
Sep 11 22:11:52.645072 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:52.644921    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06f1362b-5ccc-4ca7-8239-e5982339df97-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5\" (UID: \"06f1362b-5ccc-4ca7-8239-e5982339df97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5"
Sep 11 22:11:52.645072 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:52.644956    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnk6k\" (UniqueName: \"kubernetes.io/projected/06f1362b-5ccc-4ca7-8239-e5982339df97-kube-api-access-nnk6k\") pod \"maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5\" (UID: \"06f1362b-5ccc-4ca7-8239-e5982339df97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5"
Sep 11 22:11:52.645072 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:11:52.645044    2587 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-serving-cert: secret "maas-api-e2e-shared-a-w3-d09c96-serving-cert" not found
Sep 11 22:11:52.645224 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:11:52.645115    2587 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f1362b-5ccc-4ca7-8239-e5982339df97-maas-api-tls podName:06f1362b-5ccc-4ca7-8239-e5982339df97 nodeName:}" failed. No retries permitted until 2026-09-11 22:11:53.145098289 +0000 UTC m=+1166.320174641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/06f1362b-5ccc-4ca7-8239-e5982339df97-maas-api-tls") pod "maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5" (UID: "06f1362b-5ccc-4ca7-8239-e5982339df97") : secret "maas-api-e2e-shared-a-w3-d09c96-serving-cert" not found
Sep 11 22:11:52.645676 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:52.645648    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06f1362b-5ccc-4ca7-8239-e5982339df97-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5\" (UID: \"06f1362b-5ccc-4ca7-8239-e5982339df97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5"
Sep 11 22:11:52.647488 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:52.647459    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06f1362b-5ccc-4ca7-8239-e5982339df97-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5\" (UID: \"06f1362b-5ccc-4ca7-8239-e5982339df97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5"
Sep 11 22:11:52.653899 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:52.653875    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnk6k\" (UniqueName: \"kubernetes.io/projected/06f1362b-5ccc-4ca7-8239-e5982339df97-kube-api-access-nnk6k\") pod \"maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5\" (UID: \"06f1362b-5ccc-4ca7-8239-e5982339df97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5"
Sep 11 22:11:53.150527 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:53.150421    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06f1362b-5ccc-4ca7-8239-e5982339df97-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5\" (UID: \"06f1362b-5ccc-4ca7-8239-e5982339df97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5"
Sep 11 22:11:53.153446 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:53.153412    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06f1362b-5ccc-4ca7-8239-e5982339df97-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5\" (UID: \"06f1362b-5ccc-4ca7-8239-e5982339df97\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5"
Sep 11 22:11:53.401229 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:53.401125    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5"
Sep 11 22:11:53.682524 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:53.682488    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5"]
Sep 11 22:11:53.683992 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:11:53.683963    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f1362b_5ccc_4ca7_8239_e5982339df97.slice/crio-edfed889476adff5a846d074807661ec6c0c6e7b1c751162ad2468f9404cabda WatchSource:0}: Error finding container edfed889476adff5a846d074807661ec6c0c6e7b1c751162ad2468f9404cabda: Status 404 returned error can't find the container with id edfed889476adff5a846d074807661ec6c0c6e7b1c751162ad2468f9404cabda
Sep 11 22:11:54.647582 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:54.647533    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5" event={"ID":"06f1362b-5ccc-4ca7-8239-e5982339df97","Type":"ContainerStarted","Data":"530219b2842440a7d4e8ab03b775a06bf7e35f857c37d1112a471f8352264b06"}
Sep 11 22:11:54.647781 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:54.647609    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5" event={"ID":"06f1362b-5ccc-4ca7-8239-e5982339df97","Type":"ContainerStarted","Data":"edfed889476adff5a846d074807661ec6c0c6e7b1c751162ad2468f9404cabda"}
Sep 11 22:11:54.647781 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:54.647636    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5"
Sep 11 22:11:54.672747 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:11:54.672692    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5" podStartSLOduration=2.227948805 podStartE2EDuration="2.672676103s" totalImagesPullingTime="444.727298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:11:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:11:54.671129945 +0000 UTC m=+1167.846206336" watchObservedRunningTime="2026-09-11 22:11:54.672676103 +0000 UTC m=+1167.847752473"
Sep 11 22:12:00.659456 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:00.659418    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5"
Sep 11 22:12:01.511139 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:01.511094    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"]
Sep 11 22:12:01.511400 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:01.511371    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg" podUID="b68ba8c5-24f8-4a61-ae26-e025fa65f52f" containerName="istio-proxy" containerID="cri-o://5baffda5eb1dd95379275440961e456f754ed4b7aa8b8dc41c1103539448390f" gracePeriod=30
Sep 11 22:12:02.800075 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:02.800036    2587 patch_prober.go:28] interesting pod/e2e-noten-26d968-openshift-default-845977b986-v5hcg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 22:12:02.800507 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:02.800092    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg" podUID="b68ba8c5-24f8-4a61-ae26-e025fa65f52f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 22:12:05.281916 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:05.281879    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:12:06.720611 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.720564    2587 generic.go:356] "Generic (PLEG): container finished" podID="b68ba8c5-24f8-4a61-ae26-e025fa65f52f" containerID="5baffda5eb1dd95379275440961e456f754ed4b7aa8b8dc41c1103539448390f" exitCode=0
Sep 11 22:12:06.721011 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.720625    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg" event={"ID":"b68ba8c5-24f8-4a61-ae26-e025fa65f52f","Type":"ContainerDied","Data":"5baffda5eb1dd95379275440961e456f754ed4b7aa8b8dc41c1103539448390f"}
Sep 11 22:12:06.775157 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.775074    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:12:06.906783 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.906749    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istio-data\") pod \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") "
Sep 11 22:12:06.906783 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.906787    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-workload-socket\") pod \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") "
Sep 11 22:12:06.906987 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.906805    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istio-envoy\") pod \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") "
Sep 11 22:12:06.906987 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.906876    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istio-podinfo\") pod \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") "
Sep 11 22:12:06.906987 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.906975    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istio-token\") pod \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") "
Sep 11 22:12:06.907142 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.907013    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-workload-certs\") pod \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") "
Sep 11 22:12:06.907142 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.907022    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istio-data" (OuterVolumeSpecName: "istio-data") pod "b68ba8c5-24f8-4a61-ae26-e025fa65f52f" (UID: "b68ba8c5-24f8-4a61-ae26-e025fa65f52f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:12:06.907142 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.907035    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b68ba8c5-24f8-4a61-ae26-e025fa65f52f" (UID: "b68ba8c5-24f8-4a61-ae26-e025fa65f52f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:12:06.907142 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.907058    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istiod-ca-cert\") pod \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") "
Sep 11 22:12:06.907142 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.907105    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pj5rs\" (UniqueName: \"kubernetes.io/projected/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-kube-api-access-pj5rs\") pod \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") "
Sep 11 22:12:06.907142 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.907138    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-credential-socket\") pod \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\" (UID: \"b68ba8c5-24f8-4a61-ae26-e025fa65f52f\") "
Sep 11 22:12:06.907558 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.907191    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b68ba8c5-24f8-4a61-ae26-e025fa65f52f" (UID: "b68ba8c5-24f8-4a61-ae26-e025fa65f52f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:12:06.907558 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.907375    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b68ba8c5-24f8-4a61-ae26-e025fa65f52f" (UID: "b68ba8c5-24f8-4a61-ae26-e025fa65f52f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:12:06.907558 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.907441    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b68ba8c5-24f8-4a61-ae26-e025fa65f52f" (UID: "b68ba8c5-24f8-4a61-ae26-e025fa65f52f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:12:06.907736 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.907599    2587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-workload-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:12:06.907736 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.907619    2587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istiod-ca-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:12:06.907736 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.907632    2587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-credential-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:12:06.907736 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.907646    2587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istio-data\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:12:06.907736 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.907659    2587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-workload-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:12:06.909158 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.909125    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b68ba8c5-24f8-4a61-ae26-e025fa65f52f" (UID: "b68ba8c5-24f8-4a61-ae26-e025fa65f52f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:12:06.909158 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.909139    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-kube-api-access-pj5rs" (OuterVolumeSpecName: "kube-api-access-pj5rs") pod "b68ba8c5-24f8-4a61-ae26-e025fa65f52f" (UID: "b68ba8c5-24f8-4a61-ae26-e025fa65f52f"). InnerVolumeSpecName "kube-api-access-pj5rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:12:06.909318 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.909197    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istio-token" (OuterVolumeSpecName: "istio-token") pod "b68ba8c5-24f8-4a61-ae26-e025fa65f52f" (UID: "b68ba8c5-24f8-4a61-ae26-e025fa65f52f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:12:06.909318 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:06.909203    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b68ba8c5-24f8-4a61-ae26-e025fa65f52f" (UID: "b68ba8c5-24f8-4a61-ae26-e025fa65f52f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:12:07.008661 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:07.008623    2587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istio-envoy\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:12:07.008661 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:07.008650    2587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istio-podinfo\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:12:07.008661 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:07.008659    2587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-istio-token\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:12:07.008661 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:07.008667    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj5rs\" (UniqueName: \"kubernetes.io/projected/b68ba8c5-24f8-4a61-ae26-e025fa65f52f-kube-api-access-pj5rs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:12:07.734268 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:07.734227    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg" event={"ID":"b68ba8c5-24f8-4a61-ae26-e025fa65f52f","Type":"ContainerDied","Data":"bc8eef87148282819b685f614746d4462a8a26bace31e64635d5ebdf7024c230"}
Sep 11 22:12:07.734908 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:07.734864    2587 scope.go:117] "RemoveContainer" containerID="5baffda5eb1dd95379275440961e456f754ed4b7aa8b8dc41c1103539448390f"
Sep 11 22:12:07.735320 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:07.735302    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"
Sep 11 22:12:07.760999 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:07.760960    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"]
Sep 11 22:12:07.761182 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:07.761016    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-26d968-openshift-default-845977b986-v5hcg"]
Sep 11 22:12:09.343023 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:09.342979    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b68ba8c5-24f8-4a61-ae26-e025fa65f52f" path="/var/lib/kubelet/pods/b68ba8c5-24f8-4a61-ae26-e025fa65f52f/volumes"
Sep 11 22:12:16.798889 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:16.798851    2587 generic.go:356] "Generic (PLEG): container finished" podID="64f7d428-457a-4b96-bb21-266305d3cab7" containerID="a604650a1a4e11f913267cc4f25d3acee64b5313d8912e393d2d9ba8110c0409" exitCode=0
Sep 11 22:12:16.799428 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:16.798923    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j" event={"ID":"64f7d428-457a-4b96-bb21-266305d3cab7","Type":"ContainerDied","Data":"a604650a1a4e11f913267cc4f25d3acee64b5313d8912e393d2d9ba8110c0409"}
Sep 11 22:12:16.799428 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:16.799393    2587 scope.go:117] "RemoveContainer" containerID="a604650a1a4e11f913267cc4f25d3acee64b5313d8912e393d2d9ba8110c0409"
Sep 11 22:12:16.800587 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:16.800560    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:12:16.800661 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:16.800627    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:12:16.800699 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:16.800660    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:12:17.600163 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:17.600118    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:12:17.807071 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:17.807037    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j" event={"ID":"64f7d428-457a-4b96-bb21-266305d3cab7","Type":"ContainerStarted","Data":"c033d8f50ff1c0ff5636388ffa9e7be508fd1cdd52bfe6571b5ee1d670da3946"}
Sep 11 22:12:18.174032 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:18.173996    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:12:18.175219 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:18.175190    2587 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 11 22:12:18.175341 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:18.175242    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j" podUID="64f7d428-457a-4b96-bb21-266305d3cab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 11 22:12:19.175081 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:19.175022    2587 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 11 22:12:19.175484 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:19.175128    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j" podUID="64f7d428-457a-4b96-bb21-266305d3cab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 11 22:12:20.174172 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:20.174128    2587 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 11 22:12:20.174444 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:20.174189    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j" podUID="64f7d428-457a-4b96-bb21-266305d3cab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 11 22:12:21.174632 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:21.174595    2587 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 11 22:12:21.175069 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:21.174666    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j" podUID="64f7d428-457a-4b96-bb21-266305d3cab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 11 22:12:22.174159 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:22.174116    2587 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Sep 11 22:12:22.174369 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:22.174198    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j" podUID="64f7d428-457a-4b96-bb21-266305d3cab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Sep 11 22:12:23.178836 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:23.178799    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:12:23.179224 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:23.179194    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:12:23.179794 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:23.179772    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:12:27.098393 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:27.098354    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp"]
Sep 11 22:12:27.099021 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:27.098990    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b68ba8c5-24f8-4a61-ae26-e025fa65f52f" containerName="istio-proxy"
Sep 11 22:12:27.099021 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:27.099015    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="b68ba8c5-24f8-4a61-ae26-e025fa65f52f" containerName="istio-proxy"
Sep 11 22:12:27.099214 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:27.099122    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="b68ba8c5-24f8-4a61-ae26-e025fa65f52f" containerName="istio-proxy"
Sep 11 22:12:27.102457 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:27.102435    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp"
Sep 11 22:12:27.105282 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:27.105260    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-966527-serving-cert\""
Sep 11 22:12:27.113292 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:27.113255    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp"]
Sep 11 22:12:27.142288 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:27.142254    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b82f3465-9969-4110-88a4-dfec9d1bc7a0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp\" (UID: \"b82f3465-9969-4110-88a4-dfec9d1bc7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp"
Sep 11 22:12:27.142503 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:27.142476    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b82f3465-9969-4110-88a4-dfec9d1bc7a0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp\" (UID: \"b82f3465-9969-4110-88a4-dfec9d1bc7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp"
Sep 11 22:12:27.142562 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:27.142548    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b82f3465-9969-4110-88a4-dfec9d1bc7a0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp\" (UID: \"b82f3465-9969-4110-88a4-dfec9d1bc7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp"
Sep 11 22:12:27.142610 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:27.142595    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgpst\" (UniqueName: \"kubernetes.io/projected/b82f3465-9969-4110-88a4-dfec9d1bc7a0-kube-api-access-kgpst\") pod \"maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp\" (UID: \"b82f3465-9969-4110-88a4-dfec9d1bc7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp"
Sep 11 22:12:27.243612 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:27.243577    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b82f3465-9969-4110-88a4-dfec9d1bc7a0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp\" (UID: \"b82f3465-9969-4110-88a4-dfec9d1bc7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp"
Sep 11 22:12:27.243824 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:27.243709    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b82f3465-9969-4110-88a4-dfec9d1bc7a0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp\" (UID: \"b82f3465-9969-4110-88a4-dfec9d1bc7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp"
Sep 11 22:12:27.243824 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:27.243739    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b82f3465-9969-4110-88a4-dfec9d1bc7a0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp\" (UID: \"b82f3465-9969-4110-88a4-dfec9d1bc7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp"
Sep 11 22:12:27.243824 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:27.243766    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgpst\" (UniqueName: \"kubernetes.io/projected/b82f3465-9969-4110-88a4-dfec9d1bc7a0-kube-api-access-kgpst\") pod \"maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp\" (UID: \"b82f3465-9969-4110-88a4-dfec9d1bc7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp"
Sep 11 22:12:27.244440 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:27.244412    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b82f3465-9969-4110-88a4-dfec9d1bc7a0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp\" (UID: \"b82f3465-9969-4110-88a4-dfec9d1bc7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp"
Sep 11 22:12:27.246168 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:27.246145    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b82f3465-9969-4110-88a4-dfec9d1bc7a0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp\" (UID: \"b82f3465-9969-4110-88a4-dfec9d1bc7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp"
Sep 11 22:12:27.247654 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:27.247637    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-966527-serving-cert\""
Sep 11 22:12:27.257032 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:27.256999    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b82f3465-9969-4110-88a4-dfec9d1bc7a0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp\" (UID: \"b82f3465-9969-4110-88a4-dfec9d1bc7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp"
Sep 11 22:12:27.260454 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:27.260421    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgpst\" (UniqueName: \"kubernetes.io/projected/b82f3465-9969-4110-88a4-dfec9d1bc7a0-kube-api-access-kgpst\") pod \"maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp\" (UID: \"b82f3465-9969-4110-88a4-dfec9d1bc7a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp"
Sep 11 22:12:27.378229 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:27.378141    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw6qf_7ab8a820-c9ec-415c-bfaa-3c9744ac040a/ovn-acl-logging/0.log"
Sep 11 22:12:27.383548 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:27.383525    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw6qf_7ab8a820-c9ec-415c-bfaa-3c9744ac040a/ovn-acl-logging/0.log"
Sep 11 22:12:27.415373 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:27.415319    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp"
Sep 11 22:12:27.694880 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:27.694850    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp"]
Sep 11 22:12:27.696772 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:12:27.696740    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb82f3465_9969_4110_88a4_dfec9d1bc7a0.slice/crio-802949714e1465193ad44b82592a7e6da9efbae4c8648fd51b647c1da43c8ed7 WatchSource:0}: Error finding container 802949714e1465193ad44b82592a7e6da9efbae4c8648fd51b647c1da43c8ed7: Status 404 returned error can't find the container with id 802949714e1465193ad44b82592a7e6da9efbae4c8648fd51b647c1da43c8ed7
Sep 11 22:12:27.876157 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:27.876107    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp" event={"ID":"b82f3465-9969-4110-88a4-dfec9d1bc7a0","Type":"ContainerStarted","Data":"802949714e1465193ad44b82592a7e6da9efbae4c8648fd51b647c1da43c8ed7"}
Sep 11 22:12:28.883092 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:28.883057    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp" event={"ID":"b82f3465-9969-4110-88a4-dfec9d1bc7a0","Type":"ContainerStarted","Data":"65ca001cf5026a29612214878bcb0a15a2d56aa89049aca0d6bd7d73aa5c2323"}
Sep 11 22:12:28.883502 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:28.883128    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp"
Sep 11 22:12:28.900822 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:28.900763    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp" podStartSLOduration=1.459018035 podStartE2EDuration="1.900743763s" totalImagesPullingTime="441.725728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:12:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:12:28.899855493 +0000 UTC m=+1202.074931888" watchObservedRunningTime="2026-09-11 22:12:28.900743763 +0000 UTC m=+1202.075820133"
Sep 11 22:12:34.894540 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:34.894506    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp"
Sep 11 22:12:45.174129 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:12:45.174092    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:13:14.280813 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:14.280770    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:13:33.986531 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:33.986490    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"]
Sep 11 22:13:33.990893 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:33.990870    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:13:33.993966 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:33.993935    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d09c96\"/\"openshift-service-ca.crt\""
Sep 11 22:13:33.995289 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:33.995264    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d09c96\"/\"auth-test-model-d09c96-kserve-self-signed-certs\""
Sep 11 22:13:33.995402 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:33.995279    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d09c96\"/\"kube-root-ca.crt\""
Sep 11 22:13:33.995564 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:33.995544    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d09c96\"/\"default-dockercfg-jp7v5\""
Sep 11 22:13:34.001187 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:34.001163    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"]
Sep 11 22:13:34.056063 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:34.056023    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e795107-3b44-4fff-9d46-367022d34fc2-home\") pod \"auth-test-model-d09c96-kserve-776898dbdc-qbg46\" (UID: \"7e795107-3b44-4fff-9d46-367022d34fc2\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:13:34.056230 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:34.056077    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e795107-3b44-4fff-9d46-367022d34fc2-tls-certs\") pod \"auth-test-model-d09c96-kserve-776898dbdc-qbg46\" (UID: \"7e795107-3b44-4fff-9d46-367022d34fc2\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:13:34.056230 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:34.056158    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e795107-3b44-4fff-9d46-367022d34fc2-dshm\") pod \"auth-test-model-d09c96-kserve-776898dbdc-qbg46\" (UID: \"7e795107-3b44-4fff-9d46-367022d34fc2\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:13:34.056230 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:34.056219    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e795107-3b44-4fff-9d46-367022d34fc2-tmp-dir\") pod \"auth-test-model-d09c96-kserve-776898dbdc-qbg46\" (UID: \"7e795107-3b44-4fff-9d46-367022d34fc2\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:13:34.056385 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:34.056273    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh65x\" (UniqueName: \"kubernetes.io/projected/7e795107-3b44-4fff-9d46-367022d34fc2-kube-api-access-hh65x\") pod \"auth-test-model-d09c96-kserve-776898dbdc-qbg46\" (UID: \"7e795107-3b44-4fff-9d46-367022d34fc2\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:13:34.056440 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:34.056392    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e795107-3b44-4fff-9d46-367022d34fc2-model-cache\") pod \"auth-test-model-d09c96-kserve-776898dbdc-qbg46\" (UID: \"7e795107-3b44-4fff-9d46-367022d34fc2\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:13:34.157906 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:34.157866    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e795107-3b44-4fff-9d46-367022d34fc2-tmp-dir\") pod \"auth-test-model-d09c96-kserve-776898dbdc-qbg46\" (UID: \"7e795107-3b44-4fff-9d46-367022d34fc2\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:13:34.158062 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:34.157915    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh65x\" (UniqueName: \"kubernetes.io/projected/7e795107-3b44-4fff-9d46-367022d34fc2-kube-api-access-hh65x\") pod \"auth-test-model-d09c96-kserve-776898dbdc-qbg46\" (UID: \"7e795107-3b44-4fff-9d46-367022d34fc2\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:13:34.158062 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:34.157998    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e795107-3b44-4fff-9d46-367022d34fc2-model-cache\") pod \"auth-test-model-d09c96-kserve-776898dbdc-qbg46\" (UID: \"7e795107-3b44-4fff-9d46-367022d34fc2\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:13:34.158062 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:34.158030    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e795107-3b44-4fff-9d46-367022d34fc2-home\") pod \"auth-test-model-d09c96-kserve-776898dbdc-qbg46\" (UID: \"7e795107-3b44-4fff-9d46-367022d34fc2\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:13:34.158195 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:34.158075    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e795107-3b44-4fff-9d46-367022d34fc2-tls-certs\") pod \"auth-test-model-d09c96-kserve-776898dbdc-qbg46\" (UID: \"7e795107-3b44-4fff-9d46-367022d34fc2\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:13:34.158195 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:34.158136    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e795107-3b44-4fff-9d46-367022d34fc2-dshm\") pod \"auth-test-model-d09c96-kserve-776898dbdc-qbg46\" (UID: \"7e795107-3b44-4fff-9d46-367022d34fc2\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:13:34.158393 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:34.158369    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e795107-3b44-4fff-9d46-367022d34fc2-tmp-dir\") pod \"auth-test-model-d09c96-kserve-776898dbdc-qbg46\" (UID: \"7e795107-3b44-4fff-9d46-367022d34fc2\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:13:34.158443 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:34.158424    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e795107-3b44-4fff-9d46-367022d34fc2-model-cache\") pod \"auth-test-model-d09c96-kserve-776898dbdc-qbg46\" (UID: \"7e795107-3b44-4fff-9d46-367022d34fc2\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:13:34.158489 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:34.158443    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e795107-3b44-4fff-9d46-367022d34fc2-home\") pod \"auth-test-model-d09c96-kserve-776898dbdc-qbg46\" (UID: \"7e795107-3b44-4fff-9d46-367022d34fc2\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:13:34.160387 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:34.160365    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e795107-3b44-4fff-9d46-367022d34fc2-dshm\") pod \"auth-test-model-d09c96-kserve-776898dbdc-qbg46\" (UID: \"7e795107-3b44-4fff-9d46-367022d34fc2\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:13:34.160650 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:34.160632    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e795107-3b44-4fff-9d46-367022d34fc2-tls-certs\") pod \"auth-test-model-d09c96-kserve-776898dbdc-qbg46\" (UID: \"7e795107-3b44-4fff-9d46-367022d34fc2\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:13:34.165970 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:34.165947    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh65x\" (UniqueName: \"kubernetes.io/projected/7e795107-3b44-4fff-9d46-367022d34fc2-kube-api-access-hh65x\") pod \"auth-test-model-d09c96-kserve-776898dbdc-qbg46\" (UID: \"7e795107-3b44-4fff-9d46-367022d34fc2\") " pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:13:34.301389 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:34.301268    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:13:34.573023 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:34.572992    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"]
Sep 11 22:13:34.574630 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:13:34.574603    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e795107_3b44_4fff_9d46_367022d34fc2.slice/crio-b3a69fca5441473aafa7997f992dab4b53f03f99679272518cfad388bb798997 WatchSource:0}: Error finding container b3a69fca5441473aafa7997f992dab4b53f03f99679272518cfad388bb798997: Status 404 returned error can't find the container with id b3a69fca5441473aafa7997f992dab4b53f03f99679272518cfad388bb798997
Sep 11 22:13:35.248970 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:35.248931    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46" event={"ID":"7e795107-3b44-4fff-9d46-367022d34fc2","Type":"ContainerStarted","Data":"73d37d8d199d31e9881c80d00154e7c2a24e471ffe2783cd5ef943cb14141d82"}
Sep 11 22:13:35.248970 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:35.248974    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46" event={"ID":"7e795107-3b44-4fff-9d46-367022d34fc2","Type":"ContainerStarted","Data":"b3a69fca5441473aafa7997f992dab4b53f03f99679272518cfad388bb798997"}
Sep 11 22:13:35.267221 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:35.267168    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46" podStartSLOduration=2.267152824 podStartE2EDuration="2.267152824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:13:35.26671246 +0000 UTC m=+1268.441788849" watchObservedRunningTime="2026-09-11 22:13:35.267152824 +0000 UTC m=+1268.442229212"
Sep 11 22:13:38.372471 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:38.372433    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:13:44.301418 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:44.301370    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:13:44.301817 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:44.301437    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:13:44.314169 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:44.314137    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:13:45.325213 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:13:45.325184    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:14:03.669958 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:03.669925    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:14:24.852802 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:24.852764    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh"]
Sep 11 22:14:24.853417 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:24.853032    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh" podUID="e32e4ca9-7442-4025-974c-77c9a57878d5" containerName="maas-api" containerID="cri-o://c3c28977bee453e315187b1cd74eda2399c7c4685c2ce96ba36080ab527dc4b2" gracePeriod=30
Sep 11 22:14:25.128643 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:25.128613    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh"
Sep 11 22:14:25.189668 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:25.189625    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e32e4ca9-7442-4025-974c-77c9a57878d5-maas-api-tls\") pod \"e32e4ca9-7442-4025-974c-77c9a57878d5\" (UID: \"e32e4ca9-7442-4025-974c-77c9a57878d5\") "
Sep 11 22:14:25.190001 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:25.189978    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e32e4ca9-7442-4025-974c-77c9a57878d5-openshift-service-ca\") pod \"e32e4ca9-7442-4025-974c-77c9a57878d5\" (UID: \"e32e4ca9-7442-4025-974c-77c9a57878d5\") "
Sep 11 22:14:25.190278 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:25.190250    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bd5km\" (UniqueName: \"kubernetes.io/projected/e32e4ca9-7442-4025-974c-77c9a57878d5-kube-api-access-bd5km\") pod \"e32e4ca9-7442-4025-974c-77c9a57878d5\" (UID: \"e32e4ca9-7442-4025-974c-77c9a57878d5\") "
Sep 11 22:14:25.190605 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:25.190588    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e32e4ca9-7442-4025-974c-77c9a57878d5-metrics-certs\") pod \"e32e4ca9-7442-4025-974c-77c9a57878d5\" (UID: \"e32e4ca9-7442-4025-974c-77c9a57878d5\") "
Sep 11 22:14:25.191958 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:25.191581    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e32e4ca9-7442-4025-974c-77c9a57878d5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e32e4ca9-7442-4025-974c-77c9a57878d5" (UID: "e32e4ca9-7442-4025-974c-77c9a57878d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:14:25.191958 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:25.191908    2587 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e32e4ca9-7442-4025-974c-77c9a57878d5-openshift-service-ca\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:14:25.194528 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:25.194475    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e32e4ca9-7442-4025-974c-77c9a57878d5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e32e4ca9-7442-4025-974c-77c9a57878d5" (UID: "e32e4ca9-7442-4025-974c-77c9a57878d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:14:25.195777 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:25.195055    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e32e4ca9-7442-4025-974c-77c9a57878d5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e32e4ca9-7442-4025-974c-77c9a57878d5" (UID: "e32e4ca9-7442-4025-974c-77c9a57878d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:14:25.199443 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:25.199391    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32e4ca9-7442-4025-974c-77c9a57878d5-kube-api-access-bd5km" (OuterVolumeSpecName: "kube-api-access-bd5km") pod "e32e4ca9-7442-4025-974c-77c9a57878d5" (UID: "e32e4ca9-7442-4025-974c-77c9a57878d5"). InnerVolumeSpecName "kube-api-access-bd5km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:14:25.293102 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:25.293061    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd5km\" (UniqueName: \"kubernetes.io/projected/e32e4ca9-7442-4025-974c-77c9a57878d5-kube-api-access-bd5km\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:14:25.293102 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:25.293096    2587 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e32e4ca9-7442-4025-974c-77c9a57878d5-metrics-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:14:25.293102 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:25.293106    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e32e4ca9-7442-4025-974c-77c9a57878d5-maas-api-tls\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:14:25.546850 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:25.546809    2587 generic.go:356] "Generic (PLEG): container finished" podID="e32e4ca9-7442-4025-974c-77c9a57878d5" containerID="c3c28977bee453e315187b1cd74eda2399c7c4685c2ce96ba36080ab527dc4b2" exitCode=0
Sep 11 22:14:25.547034 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:25.546860    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh"
Sep 11 22:14:25.547034 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:25.546889    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh" event={"ID":"e32e4ca9-7442-4025-974c-77c9a57878d5","Type":"ContainerDied","Data":"c3c28977bee453e315187b1cd74eda2399c7c4685c2ce96ba36080ab527dc4b2"}
Sep 11 22:14:25.547034 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:25.546946    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh" event={"ID":"e32e4ca9-7442-4025-974c-77c9a57878d5","Type":"ContainerDied","Data":"f33690a891246e9149a10a49c634956f187e5f110da020d38f28dbfd3ed1a8e2"}
Sep 11 22:14:25.547034 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:25.546969    2587 scope.go:117] "RemoveContainer" containerID="c3c28977bee453e315187b1cd74eda2399c7c4685c2ce96ba36080ab527dc4b2"
Sep 11 22:14:25.556282 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:25.556265    2587 scope.go:117] "RemoveContainer" containerID="c3c28977bee453e315187b1cd74eda2399c7c4685c2ce96ba36080ab527dc4b2"
Sep 11 22:14:25.556590 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:14:25.556569    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3c28977bee453e315187b1cd74eda2399c7c4685c2ce96ba36080ab527dc4b2\": container with ID starting with c3c28977bee453e315187b1cd74eda2399c7c4685c2ce96ba36080ab527dc4b2 not found: ID does not exist" containerID="c3c28977bee453e315187b1cd74eda2399c7c4685c2ce96ba36080ab527dc4b2"
Sep 11 22:14:25.556640 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:25.556599    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3c28977bee453e315187b1cd74eda2399c7c4685c2ce96ba36080ab527dc4b2"} err="failed to get container status \"c3c28977bee453e315187b1cd74eda2399c7c4685c2ce96ba36080ab527dc4b2\": rpc error: code = NotFound desc = could not find container \"c3c28977bee453e315187b1cd74eda2399c7c4685c2ce96ba36080ab527dc4b2\": container with ID starting with c3c28977bee453e315187b1cd74eda2399c7c4685c2ce96ba36080ab527dc4b2 not found: ID does not exist"
Sep 11 22:14:25.564774 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:25.564747    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh"]
Sep 11 22:14:25.569112 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:25.569079    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-92d3298e-5b8f8b4cc6-ntcjh"]
Sep 11 22:14:27.338124 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:27.338091    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e32e4ca9-7442-4025-974c-77c9a57878d5" path="/var/lib/kubelet/pods/e32e4ca9-7442-4025-974c-77c9a57878d5/volumes"
Sep 11 22:14:27.531924 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:27.531893    2587 scope.go:117] "RemoveContainer" containerID="43c40e9ccadd6b7219e491b3491440d2a84466104ad0d04697262b0e961e3ccb"
Sep 11 22:14:36.193251 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:14:36.193217    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:15:01.107228 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:01.107143    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:15:17.193746 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:17.193708    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"]
Sep 11 22:15:17.194228 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:17.193979    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn" podUID="eb764f30-e047-4f61-a232-9188f54b7c04" containerName="istio-proxy" containerID="cri-o://1eae16b6e4f577c60e20e6efcaa6a5845e766e34fb9866f288211e1af3d43862" gracePeriod=30
Sep 11 22:15:18.110536 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:18.110503    2587 patch_prober.go:28] interesting pod/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 22:15:18.110700 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:18.110559    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn" podUID="eb764f30-e047-4f61-a232-9188f54b7c04" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 22:15:22.450403 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.450373    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:15:22.607736 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.607642    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k66cd\" (UniqueName: \"kubernetes.io/projected/eb764f30-e047-4f61-a232-9188f54b7c04-kube-api-access-k66cd\") pod \"eb764f30-e047-4f61-a232-9188f54b7c04\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") "
Sep 11 22:15:22.607736 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.607701    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb764f30-e047-4f61-a232-9188f54b7c04-istio-podinfo\") pod \"eb764f30-e047-4f61-a232-9188f54b7c04\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") "
Sep 11 22:15:22.607736 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.607720    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-workload-socket\") pod \"eb764f30-e047-4f61-a232-9188f54b7c04\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") "
Sep 11 22:15:22.607736 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.607738    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-workload-certs\") pod \"eb764f30-e047-4f61-a232-9188f54b7c04\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") "
Sep 11 22:15:22.608134 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.607770    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb764f30-e047-4f61-a232-9188f54b7c04-istio-token\") pod \"eb764f30-e047-4f61-a232-9188f54b7c04\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") "
Sep 11 22:15:22.608134 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.607938    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-istio-envoy\") pod \"eb764f30-e047-4f61-a232-9188f54b7c04\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") "
Sep 11 22:15:22.608134 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.607991    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb764f30-e047-4f61-a232-9188f54b7c04-istiod-ca-cert\") pod \"eb764f30-e047-4f61-a232-9188f54b7c04\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") "
Sep 11 22:15:22.608134 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.608052    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-credential-socket\") pod \"eb764f30-e047-4f61-a232-9188f54b7c04\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") "
Sep 11 22:15:22.608134 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.608089    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-istio-data\") pod \"eb764f30-e047-4f61-a232-9188f54b7c04\" (UID: \"eb764f30-e047-4f61-a232-9188f54b7c04\") "
Sep 11 22:15:22.608134 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.608085    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eb764f30-e047-4f61-a232-9188f54b7c04" (UID: "eb764f30-e047-4f61-a232-9188f54b7c04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:15:22.608465 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.608364    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eb764f30-e047-4f61-a232-9188f54b7c04" (UID: "eb764f30-e047-4f61-a232-9188f54b7c04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:15:22.608465 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.608085    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eb764f30-e047-4f61-a232-9188f54b7c04" (UID: "eb764f30-e047-4f61-a232-9188f54b7c04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:15:22.608566 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.608475    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb764f30-e047-4f61-a232-9188f54b7c04-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eb764f30-e047-4f61-a232-9188f54b7c04" (UID: "eb764f30-e047-4f61-a232-9188f54b7c04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:15:22.608566 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.608551    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-istio-data" (OuterVolumeSpecName: "istio-data") pod "eb764f30-e047-4f61-a232-9188f54b7c04" (UID: "eb764f30-e047-4f61-a232-9188f54b7c04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:15:22.608680 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.608660    2587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb764f30-e047-4f61-a232-9188f54b7c04-istiod-ca-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:15:22.608720 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.608688    2587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-credential-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:15:22.608720 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.608702    2587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-istio-data\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:15:22.608720 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.608716    2587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-workload-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:15:22.608818 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.608730    2587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-workload-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:15:22.610027 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.610004    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb764f30-e047-4f61-a232-9188f54b7c04-kube-api-access-k66cd" (OuterVolumeSpecName: "kube-api-access-k66cd") pod "eb764f30-e047-4f61-a232-9188f54b7c04" (UID: "eb764f30-e047-4f61-a232-9188f54b7c04"). InnerVolumeSpecName "kube-api-access-k66cd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:15:22.610027 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.610003    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb764f30-e047-4f61-a232-9188f54b7c04-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eb764f30-e047-4f61-a232-9188f54b7c04" (UID: "eb764f30-e047-4f61-a232-9188f54b7c04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:15:22.610232 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.610215    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb764f30-e047-4f61-a232-9188f54b7c04-istio-token" (OuterVolumeSpecName: "istio-token") pod "eb764f30-e047-4f61-a232-9188f54b7c04" (UID: "eb764f30-e047-4f61-a232-9188f54b7c04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:15:22.610382 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.610359    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eb764f30-e047-4f61-a232-9188f54b7c04" (UID: "eb764f30-e047-4f61-a232-9188f54b7c04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:15:22.709633 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.709597    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k66cd\" (UniqueName: \"kubernetes.io/projected/eb764f30-e047-4f61-a232-9188f54b7c04-kube-api-access-k66cd\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:15:22.709633 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.709625    2587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb764f30-e047-4f61-a232-9188f54b7c04-istio-podinfo\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:15:22.709633 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.709635    2587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb764f30-e047-4f61-a232-9188f54b7c04-istio-token\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:15:22.709633 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.709643    2587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb764f30-e047-4f61-a232-9188f54b7c04-istio-envoy\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:15:22.868492 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.868401    2587 generic.go:356] "Generic (PLEG): container finished" podID="eb764f30-e047-4f61-a232-9188f54b7c04" containerID="1eae16b6e4f577c60e20e6efcaa6a5845e766e34fb9866f288211e1af3d43862" exitCode=0
Sep 11 22:15:22.868492 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.868451    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn" event={"ID":"eb764f30-e047-4f61-a232-9188f54b7c04","Type":"ContainerDied","Data":"1eae16b6e4f577c60e20e6efcaa6a5845e766e34fb9866f288211e1af3d43862"}
Sep 11 22:15:22.868492 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.868458    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"
Sep 11 22:15:22.868492 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.868495    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn" event={"ID":"eb764f30-e047-4f61-a232-9188f54b7c04","Type":"ContainerDied","Data":"04c9f57940cf84896a83f7c887eb3f78a33c0d47017257a24c3d280d1632bde4"}
Sep 11 22:15:22.868810 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.868516    2587 scope.go:117] "RemoveContainer" containerID="1eae16b6e4f577c60e20e6efcaa6a5845e766e34fb9866f288211e1af3d43862"
Sep 11 22:15:22.878210 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.878186    2587 scope.go:117] "RemoveContainer" containerID="3128b007868ff5cf480df4ec024286ad26c700a81a7619f22484ef7cb064a841"
Sep 11 22:15:22.887369 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.887351    2587 scope.go:117] "RemoveContainer" containerID="1eae16b6e4f577c60e20e6efcaa6a5845e766e34fb9866f288211e1af3d43862"
Sep 11 22:15:22.887679 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:15:22.887660    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eae16b6e4f577c60e20e6efcaa6a5845e766e34fb9866f288211e1af3d43862\": container with ID starting with 1eae16b6e4f577c60e20e6efcaa6a5845e766e34fb9866f288211e1af3d43862 not found: ID does not exist" containerID="1eae16b6e4f577c60e20e6efcaa6a5845e766e34fb9866f288211e1af3d43862"
Sep 11 22:15:22.887737 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.887690    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eae16b6e4f577c60e20e6efcaa6a5845e766e34fb9866f288211e1af3d43862"} err="failed to get container status \"1eae16b6e4f577c60e20e6efcaa6a5845e766e34fb9866f288211e1af3d43862\": rpc error: code = NotFound desc = could not find container \"1eae16b6e4f577c60e20e6efcaa6a5845e766e34fb9866f288211e1af3d43862\": container with ID starting with 1eae16b6e4f577c60e20e6efcaa6a5845e766e34fb9866f288211e1af3d43862 not found: ID does not exist"
Sep 11 22:15:22.887737 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.887708    2587 scope.go:117] "RemoveContainer" containerID="3128b007868ff5cf480df4ec024286ad26c700a81a7619f22484ef7cb064a841"
Sep 11 22:15:22.887980 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:15:22.887961    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3128b007868ff5cf480df4ec024286ad26c700a81a7619f22484ef7cb064a841\": container with ID starting with 3128b007868ff5cf480df4ec024286ad26c700a81a7619f22484ef7cb064a841 not found: ID does not exist" containerID="3128b007868ff5cf480df4ec024286ad26c700a81a7619f22484ef7cb064a841"
Sep 11 22:15:22.888045 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.887989    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3128b007868ff5cf480df4ec024286ad26c700a81a7619f22484ef7cb064a841"} err="failed to get container status \"3128b007868ff5cf480df4ec024286ad26c700a81a7619f22484ef7cb064a841\": rpc error: code = NotFound desc = could not find container \"3128b007868ff5cf480df4ec024286ad26c700a81a7619f22484ef7cb064a841\": container with ID starting with 3128b007868ff5cf480df4ec024286ad26c700a81a7619f22484ef7cb064a841 not found: ID does not exist"
Sep 11 22:15:22.892070 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.892046    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"]
Sep 11 22:15:22.895894 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:22.895871    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-92d3298e-openshift-default-78b4fd7c9b-7bptn"]
Sep 11 22:15:23.336383 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:23.336346    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb764f30-e047-4f61-a232-9188f54b7c04" path="/var/lib/kubelet/pods/eb764f30-e047-4f61-a232-9188f54b7c04/volumes"
Sep 11 22:15:26.888687 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:15:26.888652    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:16:08.075279 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:08.075230    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:16:12.157795 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.157728    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"]
Sep 11 22:16:12.158315 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.158222    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb764f30-e047-4f61-a232-9188f54b7c04" containerName="istio-proxy"
Sep 11 22:16:12.158315 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.158234    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb764f30-e047-4f61-a232-9188f54b7c04" containerName="istio-proxy"
Sep 11 22:16:12.158315 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.158243    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb764f30-e047-4f61-a232-9188f54b7c04" containerName="istio-proxy"
Sep 11 22:16:12.158315 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.158248    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb764f30-e047-4f61-a232-9188f54b7c04" containerName="istio-proxy"
Sep 11 22:16:12.158315 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.158278    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e32e4ca9-7442-4025-974c-77c9a57878d5" containerName="maas-api"
Sep 11 22:16:12.158315 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.158286    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="e32e4ca9-7442-4025-974c-77c9a57878d5" containerName="maas-api"
Sep 11 22:16:12.158942 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.158363    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="e32e4ca9-7442-4025-974c-77c9a57878d5" containerName="maas-api"
Sep 11 22:16:12.158942 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.158375    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb764f30-e047-4f61-a232-9188f54b7c04" containerName="istio-proxy"
Sep 11 22:16:12.158942 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.158382    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb764f30-e047-4f61-a232-9188f54b7c04" containerName="istio-proxy"
Sep 11 22:16:12.161863 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.161839    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.165004 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.164979    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-fe6196-openshift-default-dockercfg-ktgxp\""
Sep 11 22:16:12.174267 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.174245    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"]
Sep 11 22:16:12.232536 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.232504    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istio-envoy\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.232706 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.232545    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-workload-socket\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.232706 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.232584    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-workload-certs\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.232706 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.232614    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istio-token\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.232706 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.232660    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istiod-ca-cert\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.232706 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.232680    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istio-data\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.232706 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.232696    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-credential-socket\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.232951 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.232762    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istio-podinfo\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.232951 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.232802    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znfs6\" (UniqueName: \"kubernetes.io/projected/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-kube-api-access-znfs6\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.333806 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.333769    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istio-podinfo\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.334001 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.333833    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znfs6\" (UniqueName: \"kubernetes.io/projected/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-kube-api-access-znfs6\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.334001 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.333879    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istio-envoy\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.334001 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.333903    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-workload-socket\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.334001 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.333941    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-workload-certs\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.334001 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.333966    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istio-token\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.334264 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.334021    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istiod-ca-cert\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.334264 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.334051    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istio-data\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.334264 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.334079    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-credential-socket\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.334441 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.334390    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-workload-socket\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.334506 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.334454    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-workload-certs\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.334566 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.334542    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istio-data\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.334700 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.334679    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-credential-socket\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.334942 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.334923    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istiod-ca-cert\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.336690 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.336672    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istio-envoy\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.336966 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.336935    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istio-podinfo\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.343980 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.343952    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istio-token\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.344196 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.344176    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znfs6\" (UniqueName: \"kubernetes.io/projected/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-kube-api-access-znfs6\") pod \"e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") " pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.474931 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.474899    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:12.757159 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.757133    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"]
Sep 11 22:16:12.758890 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:16:12.758859    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd42dd49d_9cc0_438a_a871_40e72a5e2e4e.slice/crio-85cd08db8c30b1e5a81e96ca87cdb4175cb36b59770e4a4f49c381ac8128b5e5 WatchSource:0}: Error finding container 85cd08db8c30b1e5a81e96ca87cdb4175cb36b59770e4a4f49c381ac8128b5e5: Status 404 returned error can't find the container with id 85cd08db8c30b1e5a81e96ca87cdb4175cb36b59770e4a4f49c381ac8128b5e5
Sep 11 22:16:12.761636 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.761602    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:16:12.761733 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.761670    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:16:12.761733 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:12.761698    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:16:13.157102 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:13.157016    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n" event={"ID":"d42dd49d-9cc0-438a-a871-40e72a5e2e4e","Type":"ContainerStarted","Data":"0e40e1ef489677ae8f1706d5516fc800d39fa052fe8f45fee91ed66fb780afc9"}
Sep 11 22:16:13.157102 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:13.157072    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n" event={"ID":"d42dd49d-9cc0-438a-a871-40e72a5e2e4e","Type":"ContainerStarted","Data":"85cd08db8c30b1e5a81e96ca87cdb4175cb36b59770e4a4f49c381ac8128b5e5"}
Sep 11 22:16:13.179728 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:13.179674    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n" podStartSLOduration=1.179658446 podStartE2EDuration="1.179658446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:16:13.177839511 +0000 UTC m=+1426.352915910" watchObservedRunningTime="2026-09-11 22:16:13.179658446 +0000 UTC m=+1426.354734815"
Sep 11 22:16:13.475415 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:13.475376    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:13.480765 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:13.480740    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:14.169645 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:14.169607    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:14.170576 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:14.170553    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:20.373906 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:20.373821    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4"]
Sep 11 22:16:20.379792 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:20.379766    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4"
Sep 11 22:16:20.382716 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:20.382688    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-fe6196-serving-cert\""
Sep 11 22:16:20.387401 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:20.387371    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4"]
Sep 11 22:16:20.526177 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:20.526136    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c66d78e8-8d56-4707-8de4-d7551dc143e8-openshift-service-ca\") pod \"maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4\" (UID: \"c66d78e8-8d56-4707-8de4-d7551dc143e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4"
Sep 11 22:16:20.526177 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:20.526179    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c66d78e8-8d56-4707-8de4-d7551dc143e8-metrics-certs\") pod \"maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4\" (UID: \"c66d78e8-8d56-4707-8de4-d7551dc143e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4"
Sep 11 22:16:20.526457 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:20.526315    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c66d78e8-8d56-4707-8de4-d7551dc143e8-maas-api-tls\") pod \"maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4\" (UID: \"c66d78e8-8d56-4707-8de4-d7551dc143e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4"
Sep 11 22:16:20.526457 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:20.526374    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9vq4\" (UniqueName: \"kubernetes.io/projected/c66d78e8-8d56-4707-8de4-d7551dc143e8-kube-api-access-f9vq4\") pod \"maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4\" (UID: \"c66d78e8-8d56-4707-8de4-d7551dc143e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4"
Sep 11 22:16:20.627526 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:20.627446    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9vq4\" (UniqueName: \"kubernetes.io/projected/c66d78e8-8d56-4707-8de4-d7551dc143e8-kube-api-access-f9vq4\") pod \"maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4\" (UID: \"c66d78e8-8d56-4707-8de4-d7551dc143e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4"
Sep 11 22:16:20.627683 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:20.627540    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c66d78e8-8d56-4707-8de4-d7551dc143e8-openshift-service-ca\") pod \"maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4\" (UID: \"c66d78e8-8d56-4707-8de4-d7551dc143e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4"
Sep 11 22:16:20.627683 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:20.627578    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c66d78e8-8d56-4707-8de4-d7551dc143e8-metrics-certs\") pod \"maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4\" (UID: \"c66d78e8-8d56-4707-8de4-d7551dc143e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4"
Sep 11 22:16:20.627683 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:20.627659    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c66d78e8-8d56-4707-8de4-d7551dc143e8-maas-api-tls\") pod \"maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4\" (UID: \"c66d78e8-8d56-4707-8de4-d7551dc143e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4"
Sep 11 22:16:20.628320 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:20.628297    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c66d78e8-8d56-4707-8de4-d7551dc143e8-openshift-service-ca\") pod \"maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4\" (UID: \"c66d78e8-8d56-4707-8de4-d7551dc143e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4"
Sep 11 22:16:20.630099 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:20.630076    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c66d78e8-8d56-4707-8de4-d7551dc143e8-metrics-certs\") pod \"maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4\" (UID: \"c66d78e8-8d56-4707-8de4-d7551dc143e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4"
Sep 11 22:16:20.630186 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:20.630101    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c66d78e8-8d56-4707-8de4-d7551dc143e8-maas-api-tls\") pod \"maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4\" (UID: \"c66d78e8-8d56-4707-8de4-d7551dc143e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4"
Sep 11 22:16:20.635750 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:20.635728    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9vq4\" (UniqueName: \"kubernetes.io/projected/c66d78e8-8d56-4707-8de4-d7551dc143e8-kube-api-access-f9vq4\") pod \"maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4\" (UID: \"c66d78e8-8d56-4707-8de4-d7551dc143e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4"
Sep 11 22:16:20.696119 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:20.696084    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4"
Sep 11 22:16:20.974586 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:20.974551    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4"]
Sep 11 22:16:20.976106 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:16:20.976078    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66d78e8_8d56_4707_8de4_d7551dc143e8.slice/crio-fe9913a7245e2073d4ce65ef7e22aadfb75ba8e1524663efe31a10c7730e0a15 WatchSource:0}: Error finding container fe9913a7245e2073d4ce65ef7e22aadfb75ba8e1524663efe31a10c7730e0a15: Status 404 returned error can't find the container with id fe9913a7245e2073d4ce65ef7e22aadfb75ba8e1524663efe31a10c7730e0a15
Sep 11 22:16:20.977623 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:20.977608    2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 22:16:21.209307 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:21.209277    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4" event={"ID":"c66d78e8-8d56-4707-8de4-d7551dc143e8","Type":"ContainerStarted","Data":"fe9913a7245e2073d4ce65ef7e22aadfb75ba8e1524663efe31a10c7730e0a15"}
Sep 11 22:16:22.217633 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:22.217596    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4" event={"ID":"c66d78e8-8d56-4707-8de4-d7551dc143e8","Type":"ContainerStarted","Data":"92f719dcc94bafb4cd84e4c358d8228243eeefb1888b353748a97aeb0d2c532a"}
Sep 11 22:16:22.218016 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:22.217657    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4"
Sep 11 22:16:22.235075 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:22.235019    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4" podStartSLOduration=1.742141956 podStartE2EDuration="2.234999562s" totalImagesPullingTime="492.857606ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:16:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:16:22.234226769 +0000 UTC m=+1435.409303159" watchObservedRunningTime="2026-09-11 22:16:22.234999562 +0000 UTC m=+1435.410075934"
Sep 11 22:16:28.230192 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:28.230157    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4"
Sep 11 22:16:45.948218 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:45.948181    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"]
Sep 11 22:16:45.952284 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:45.952264    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:45.955285 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:45.955265    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2715d7-openshift-default-dockercfg-8446x\""
Sep 11 22:16:45.970640 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:45.970609    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"]
Sep 11 22:16:46.102212 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.102170    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istio-podinfo\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.102436 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.102223    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-workload-certs\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.102436 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.102273    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istio-data\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.102436 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.102293    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istio-token\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.102436 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.102365    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istiod-ca-cert\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.102663 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.102441    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkcxw\" (UniqueName: \"kubernetes.io/projected/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-kube-api-access-rkcxw\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.102663 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.102464    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-workload-socket\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.102663 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.102487    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istio-envoy\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.102663 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.102505    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-credential-socket\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.204059 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.203970    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istio-data\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.204059 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.204021    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istio-token\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.204288 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.204061    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istiod-ca-cert\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.204288 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.204133    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkcxw\" (UniqueName: \"kubernetes.io/projected/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-kube-api-access-rkcxw\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.204288 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.204167    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-workload-socket\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.204288 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.204204    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istio-envoy\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.204542 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.204385    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-credential-socket\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.204542 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.204456    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istio-data\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.204542 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.204508    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istio-podinfo\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.204694 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.204562    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-workload-socket\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.204694 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.204576    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-workload-certs\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.204800 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.204723    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-credential-socket\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.204858 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.204817    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-workload-certs\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.204915 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.204860    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istiod-ca-cert\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.206819 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.206798    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istio-envoy\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.206936 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.206825    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istio-podinfo\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.212211 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.212180    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istio-token\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.212304 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.212232    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkcxw\" (UniqueName: \"kubernetes.io/projected/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-kube-api-access-rkcxw\") pod \"e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") " pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.264570 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.264534    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:46.384529 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.384491    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:16:46.558341 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.558305    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"]
Sep 11 22:16:46.560390 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:16:46.560358    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a830dd_b1fe_4c3a_8eb7_c102b2fa05e7.slice/crio-edf72783fed5e18f5d8d471c1417017eb10c6e0ce16262f682e85e9372facb97 WatchSource:0}: Error finding container edf72783fed5e18f5d8d471c1417017eb10c6e0ce16262f682e85e9372facb97: Status 404 returned error can't find the container with id edf72783fed5e18f5d8d471c1417017eb10c6e0ce16262f682e85e9372facb97
Sep 11 22:16:46.563117 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.563087    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:16:46.563235 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.563165    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:16:46.563235 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:46.563209    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:16:47.370390 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:47.370354    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r" event={"ID":"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7","Type":"ContainerStarted","Data":"66e5cfa99aa777bd25ed932d9561c3226356e1df6b2254846497cba730cbbf0f"}
Sep 11 22:16:47.370840 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:47.370406    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r" event={"ID":"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7","Type":"ContainerStarted","Data":"edf72783fed5e18f5d8d471c1417017eb10c6e0ce16262f682e85e9372facb97"}
Sep 11 22:16:47.394738 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:47.394689    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r" podStartSLOduration=2.394650766 podStartE2EDuration="2.394650766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:16:47.392534621 +0000 UTC m=+1460.567611012" watchObservedRunningTime="2026-09-11 22:16:47.394650766 +0000 UTC m=+1460.569727135"
Sep 11 22:16:48.265731 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:48.265684    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:48.272050 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:48.272020    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:48.379403 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:48.379368    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:48.380210 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:48.380187    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:16:58.440522 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:58.440489    2587 generic.go:356] "Generic (PLEG): container finished" podID="d42dd49d-9cc0-438a-a871-40e72a5e2e4e" containerID="0e40e1ef489677ae8f1706d5516fc800d39fa052fe8f45fee91ed66fb780afc9" exitCode=0
Sep 11 22:16:58.440939 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:58.440563    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n" event={"ID":"d42dd49d-9cc0-438a-a871-40e72a5e2e4e","Type":"ContainerDied","Data":"0e40e1ef489677ae8f1706d5516fc800d39fa052fe8f45fee91ed66fb780afc9"}
Sep 11 22:16:58.441065 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:58.441048    2587 scope.go:117] "RemoveContainer" containerID="0e40e1ef489677ae8f1706d5516fc800d39fa052fe8f45fee91ed66fb780afc9"
Sep 11 22:16:58.442459 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:58.442430    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:16:58.442536 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:58.442496    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:16:58.442536 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:58.442529    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:16:59.170128 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:59.170090    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:59.449305 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:59.449223    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n" event={"ID":"d42dd49d-9cc0-438a-a871-40e72a5e2e4e","Type":"ContainerStarted","Data":"77e0e6256ae1e19aa690ae7e9c2e1ec531437b526a1d333cd4f0dd8241b4a508"}
Sep 11 22:16:59.475718 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:59.475674    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:16:59.478004 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:59.477373    2587 patch_prober.go:28] interesting pod/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 11 22:16:59.478004 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:16:59.477448    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n" podUID="d42dd49d-9cc0-438a-a871-40e72a5e2e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 11 22:17:00.476090 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:00.476055    2587 patch_prober.go:28] interesting pod/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 11 22:17:00.476481 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:00.476116    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n" podUID="d42dd49d-9cc0-438a-a871-40e72a5e2e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 11 22:17:01.475421 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:01.475386    2587 patch_prober.go:28] interesting pod/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 11 22:17:01.475604 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:01.475445    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n" podUID="d42dd49d-9cc0-438a-a871-40e72a5e2e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 11 22:17:02.475692 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:02.475643    2587 patch_prober.go:28] interesting pod/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 11 22:17:02.476105 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:02.475721    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n" podUID="d42dd49d-9cc0-438a-a871-40e72a5e2e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 11 22:17:03.475990 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:03.475950    2587 patch_prober.go:28] interesting pod/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 11 22:17:03.476668 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:03.476008    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n" podUID="d42dd49d-9cc0-438a-a871-40e72a5e2e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 11 22:17:04.475529 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:04.475491    2587 patch_prober.go:28] interesting pod/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 11 22:17:04.475717 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:04.475569    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n" podUID="d42dd49d-9cc0-438a-a871-40e72a5e2e4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 11 22:17:05.480444 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:05.480415    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:17:05.480951 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:05.480930    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:17:05.481689 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:05.481666    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:17:10.525116 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:10.525079    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6"]
Sep 11 22:17:10.525613 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:10.525372    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6" podUID="54e64fba-6ece-4dee-8e5f-76210f2c4ed8" containerName="maas-api" containerID="cri-o://0c4be844dc02c1cd4e21acea6a8eb88da230f85ae69411e4db8ffb4104b93dd3" gracePeriod=30
Sep 11 22:17:10.721497 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:10.721436    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6" podUID="54e64fba-6ece-4dee-8e5f-76210f2c4ed8" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.49:8443/health\": dial tcp 10.134.0.49:8443: connect: connection refused"
Sep 11 22:17:11.027205 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:11.027166    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6"
Sep 11 22:17:11.091723 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:11.091636    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8jv2\" (UniqueName: \"kubernetes.io/projected/54e64fba-6ece-4dee-8e5f-76210f2c4ed8-kube-api-access-q8jv2\") pod \"54e64fba-6ece-4dee-8e5f-76210f2c4ed8\" (UID: \"54e64fba-6ece-4dee-8e5f-76210f2c4ed8\") "
Sep 11 22:17:11.091723 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:11.091700    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e64fba-6ece-4dee-8e5f-76210f2c4ed8-openshift-service-ca\") pod \"54e64fba-6ece-4dee-8e5f-76210f2c4ed8\" (UID: \"54e64fba-6ece-4dee-8e5f-76210f2c4ed8\") "
Sep 11 22:17:11.091914 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:11.091728    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54e64fba-6ece-4dee-8e5f-76210f2c4ed8-metrics-certs\") pod \"54e64fba-6ece-4dee-8e5f-76210f2c4ed8\" (UID: \"54e64fba-6ece-4dee-8e5f-76210f2c4ed8\") "
Sep 11 22:17:11.091914 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:11.091825    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54e64fba-6ece-4dee-8e5f-76210f2c4ed8-maas-api-tls\") pod \"54e64fba-6ece-4dee-8e5f-76210f2c4ed8\" (UID: \"54e64fba-6ece-4dee-8e5f-76210f2c4ed8\") "
Sep 11 22:17:11.092147 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:11.092115    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54e64fba-6ece-4dee-8e5f-76210f2c4ed8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "54e64fba-6ece-4dee-8e5f-76210f2c4ed8" (UID: "54e64fba-6ece-4dee-8e5f-76210f2c4ed8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:17:11.092346 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:11.092303    2587 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54e64fba-6ece-4dee-8e5f-76210f2c4ed8-openshift-service-ca\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:11.093980 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:11.093957    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e64fba-6ece-4dee-8e5f-76210f2c4ed8-kube-api-access-q8jv2" (OuterVolumeSpecName: "kube-api-access-q8jv2") pod "54e64fba-6ece-4dee-8e5f-76210f2c4ed8" (UID: "54e64fba-6ece-4dee-8e5f-76210f2c4ed8"). InnerVolumeSpecName "kube-api-access-q8jv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:17:11.094053 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:11.094009    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54e64fba-6ece-4dee-8e5f-76210f2c4ed8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "54e64fba-6ece-4dee-8e5f-76210f2c4ed8" (UID: "54e64fba-6ece-4dee-8e5f-76210f2c4ed8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:17:11.094053 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:11.094038    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54e64fba-6ece-4dee-8e5f-76210f2c4ed8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "54e64fba-6ece-4dee-8e5f-76210f2c4ed8" (UID: "54e64fba-6ece-4dee-8e5f-76210f2c4ed8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:17:11.192984 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:11.192937    2587 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54e64fba-6ece-4dee-8e5f-76210f2c4ed8-metrics-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:11.192984 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:11.192974    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54e64fba-6ece-4dee-8e5f-76210f2c4ed8-maas-api-tls\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:11.192984 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:11.192988    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8jv2\" (UniqueName: \"kubernetes.io/projected/54e64fba-6ece-4dee-8e5f-76210f2c4ed8-kube-api-access-q8jv2\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:11.523038 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:11.522993    2587 generic.go:356] "Generic (PLEG): container finished" podID="54e64fba-6ece-4dee-8e5f-76210f2c4ed8" containerID="0c4be844dc02c1cd4e21acea6a8eb88da230f85ae69411e4db8ffb4104b93dd3" exitCode=0
Sep 11 22:17:11.523247 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:11.523080    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6"
Sep 11 22:17:11.523247 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:11.523085    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6" event={"ID":"54e64fba-6ece-4dee-8e5f-76210f2c4ed8","Type":"ContainerDied","Data":"0c4be844dc02c1cd4e21acea6a8eb88da230f85ae69411e4db8ffb4104b93dd3"}
Sep 11 22:17:11.523247 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:11.523150    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6" event={"ID":"54e64fba-6ece-4dee-8e5f-76210f2c4ed8","Type":"ContainerDied","Data":"0f256e74392aac074aef81b1c1eb3998df2be2193fef27309c34c257abde5049"}
Sep 11 22:17:11.523247 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:11.523178    2587 scope.go:117] "RemoveContainer" containerID="0c4be844dc02c1cd4e21acea6a8eb88da230f85ae69411e4db8ffb4104b93dd3"
Sep 11 22:17:11.533509 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:11.533308    2587 scope.go:117] "RemoveContainer" containerID="0c4be844dc02c1cd4e21acea6a8eb88da230f85ae69411e4db8ffb4104b93dd3"
Sep 11 22:17:11.533786 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:17:11.533702    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c4be844dc02c1cd4e21acea6a8eb88da230f85ae69411e4db8ffb4104b93dd3\": container with ID starting with 0c4be844dc02c1cd4e21acea6a8eb88da230f85ae69411e4db8ffb4104b93dd3 not found: ID does not exist" containerID="0c4be844dc02c1cd4e21acea6a8eb88da230f85ae69411e4db8ffb4104b93dd3"
Sep 11 22:17:11.533786 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:11.533733    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c4be844dc02c1cd4e21acea6a8eb88da230f85ae69411e4db8ffb4104b93dd3"} err="failed to get container status \"0c4be844dc02c1cd4e21acea6a8eb88da230f85ae69411e4db8ffb4104b93dd3\": rpc error: code = NotFound desc = could not find container \"0c4be844dc02c1cd4e21acea6a8eb88da230f85ae69411e4db8ffb4104b93dd3\": container with ID starting with 0c4be844dc02c1cd4e21acea6a8eb88da230f85ae69411e4db8ffb4104b93dd3 not found: ID does not exist"
Sep 11 22:17:11.541766 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:11.541742    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6"]
Sep 11 22:17:11.549062 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:11.549026    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-95d6c2-6fd669878d-pqvw6"]
Sep 11 22:17:13.337317 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:13.337281    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54e64fba-6ece-4dee-8e5f-76210f2c4ed8" path="/var/lib/kubelet/pods/54e64fba-6ece-4dee-8e5f-76210f2c4ed8/volumes"
Sep 11 22:17:16.385350 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:16.379320    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:17:17.745319 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:17.745286    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"]
Sep 11 22:17:17.745764 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:17.745552    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk" podUID="9fe4daef-73cf-4488-acec-a2b20a88db5e" containerName="istio-proxy" containerID="cri-o://7d84bf3849690074d61886b832028800f52a0377d98d8b50543c6d4062db6a5a" gracePeriod=30
Sep 11 22:17:23.055208 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.055175    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:17:23.145172 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.143705    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-istio-data\") pod \"9fe4daef-73cf-4488-acec-a2b20a88db5e\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") "
Sep 11 22:17:23.145172 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.143769    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-credential-socket\") pod \"9fe4daef-73cf-4488-acec-a2b20a88db5e\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") "
Sep 11 22:17:23.145172 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.143805    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe4daef-73cf-4488-acec-a2b20a88db5e-istio-podinfo\") pod \"9fe4daef-73cf-4488-acec-a2b20a88db5e\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") "
Sep 11 22:17:23.145172 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.143855    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-workload-socket\") pod \"9fe4daef-73cf-4488-acec-a2b20a88db5e\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") "
Sep 11 22:17:23.145172 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.143922    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sm5hv\" (UniqueName: \"kubernetes.io/projected/9fe4daef-73cf-4488-acec-a2b20a88db5e-kube-api-access-sm5hv\") pod \"9fe4daef-73cf-4488-acec-a2b20a88db5e\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") "
Sep 11 22:17:23.145172 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.143954    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fe4daef-73cf-4488-acec-a2b20a88db5e-istio-token\") pod \"9fe4daef-73cf-4488-acec-a2b20a88db5e\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") "
Sep 11 22:17:23.145172 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.144078    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-workload-certs\") pod \"9fe4daef-73cf-4488-acec-a2b20a88db5e\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") "
Sep 11 22:17:23.145172 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.144103    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-istio-envoy\") pod \"9fe4daef-73cf-4488-acec-a2b20a88db5e\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") "
Sep 11 22:17:23.145172 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.144146    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe4daef-73cf-4488-acec-a2b20a88db5e-istiod-ca-cert\") pod \"9fe4daef-73cf-4488-acec-a2b20a88db5e\" (UID: \"9fe4daef-73cf-4488-acec-a2b20a88db5e\") "
Sep 11 22:17:23.145172 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.144778    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9fe4daef-73cf-4488-acec-a2b20a88db5e" (UID: "9fe4daef-73cf-4488-acec-a2b20a88db5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:23.145172 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.144964    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-istio-data" (OuterVolumeSpecName: "istio-data") pod "9fe4daef-73cf-4488-acec-a2b20a88db5e" (UID: "9fe4daef-73cf-4488-acec-a2b20a88db5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:23.145172 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.145111    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9fe4daef-73cf-4488-acec-a2b20a88db5e" (UID: "9fe4daef-73cf-4488-acec-a2b20a88db5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:23.146043 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.145569    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9fe4daef-73cf-4488-acec-a2b20a88db5e" (UID: "9fe4daef-73cf-4488-acec-a2b20a88db5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:23.146407 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.146349    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe4daef-73cf-4488-acec-a2b20a88db5e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9fe4daef-73cf-4488-acec-a2b20a88db5e" (UID: "9fe4daef-73cf-4488-acec-a2b20a88db5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:17:23.147874 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.147841    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9fe4daef-73cf-4488-acec-a2b20a88db5e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9fe4daef-73cf-4488-acec-a2b20a88db5e" (UID: "9fe4daef-73cf-4488-acec-a2b20a88db5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:17:23.148269 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.148234    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe4daef-73cf-4488-acec-a2b20a88db5e-kube-api-access-sm5hv" (OuterVolumeSpecName: "kube-api-access-sm5hv") pod "9fe4daef-73cf-4488-acec-a2b20a88db5e" (UID: "9fe4daef-73cf-4488-acec-a2b20a88db5e"). InnerVolumeSpecName "kube-api-access-sm5hv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:17:23.148409 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.148377    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe4daef-73cf-4488-acec-a2b20a88db5e-istio-token" (OuterVolumeSpecName: "istio-token") pod "9fe4daef-73cf-4488-acec-a2b20a88db5e" (UID: "9fe4daef-73cf-4488-acec-a2b20a88db5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:17:23.149036 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.149009    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9fe4daef-73cf-4488-acec-a2b20a88db5e" (UID: "9fe4daef-73cf-4488-acec-a2b20a88db5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:23.245480 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.245434    2587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-istio-data\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:23.245480 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.245481    2587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-credential-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:23.245762 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.245496    2587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9fe4daef-73cf-4488-acec-a2b20a88db5e-istio-podinfo\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:23.245762 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.245509    2587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-workload-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:23.245762 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.245521    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm5hv\" (UniqueName: \"kubernetes.io/projected/9fe4daef-73cf-4488-acec-a2b20a88db5e-kube-api-access-sm5hv\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:23.245762 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.245533    2587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9fe4daef-73cf-4488-acec-a2b20a88db5e-istio-token\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:23.245762 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.245548    2587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-workload-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:23.245762 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.245560    2587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9fe4daef-73cf-4488-acec-a2b20a88db5e-istio-envoy\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:23.245762 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.245574    2587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9fe4daef-73cf-4488-acec-a2b20a88db5e-istiod-ca-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:23.596359 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.596306    2587 generic.go:356] "Generic (PLEG): container finished" podID="9fe4daef-73cf-4488-acec-a2b20a88db5e" containerID="7d84bf3849690074d61886b832028800f52a0377d98d8b50543c6d4062db6a5a" exitCode=0
Sep 11 22:17:23.596527 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.596366    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"
Sep 11 22:17:23.596527 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.596393    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk" event={"ID":"9fe4daef-73cf-4488-acec-a2b20a88db5e","Type":"ContainerDied","Data":"7d84bf3849690074d61886b832028800f52a0377d98d8b50543c6d4062db6a5a"}
Sep 11 22:17:23.596527 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.596450    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk" event={"ID":"9fe4daef-73cf-4488-acec-a2b20a88db5e","Type":"ContainerDied","Data":"d363556873a292a1abf6b40f87c223ce9201e488f403e86f4c6f1f543af02a44"}
Sep 11 22:17:23.596527 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.596474    2587 scope.go:117] "RemoveContainer" containerID="7d84bf3849690074d61886b832028800f52a0377d98d8b50543c6d4062db6a5a"
Sep 11 22:17:23.609731 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.609689    2587 scope.go:117] "RemoveContainer" containerID="1d9b78d1272c6daf4a1dfb776ce66888821506805c8d7a20cf42c85144f0751b"
Sep 11 22:17:23.619185 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.619159    2587 scope.go:117] "RemoveContainer" containerID="7d84bf3849690074d61886b832028800f52a0377d98d8b50543c6d4062db6a5a"
Sep 11 22:17:23.622710 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:17:23.622583    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d84bf3849690074d61886b832028800f52a0377d98d8b50543c6d4062db6a5a\": container with ID starting with 7d84bf3849690074d61886b832028800f52a0377d98d8b50543c6d4062db6a5a not found: ID does not exist" containerID="7d84bf3849690074d61886b832028800f52a0377d98d8b50543c6d4062db6a5a"
Sep 11 22:17:23.622710 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.622626    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d84bf3849690074d61886b832028800f52a0377d98d8b50543c6d4062db6a5a"} err="failed to get container status \"7d84bf3849690074d61886b832028800f52a0377d98d8b50543c6d4062db6a5a\": rpc error: code = NotFound desc = could not find container \"7d84bf3849690074d61886b832028800f52a0377d98d8b50543c6d4062db6a5a\": container with ID starting with 7d84bf3849690074d61886b832028800f52a0377d98d8b50543c6d4062db6a5a not found: ID does not exist"
Sep 11 22:17:23.622710 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.622651    2587 scope.go:117] "RemoveContainer" containerID="1d9b78d1272c6daf4a1dfb776ce66888821506805c8d7a20cf42c85144f0751b"
Sep 11 22:17:23.631771 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:17:23.631730    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d9b78d1272c6daf4a1dfb776ce66888821506805c8d7a20cf42c85144f0751b\": container with ID starting with 1d9b78d1272c6daf4a1dfb776ce66888821506805c8d7a20cf42c85144f0751b not found: ID does not exist" containerID="1d9b78d1272c6daf4a1dfb776ce66888821506805c8d7a20cf42c85144f0751b"
Sep 11 22:17:23.631920 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.631777    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d9b78d1272c6daf4a1dfb776ce66888821506805c8d7a20cf42c85144f0751b"} err="failed to get container status \"1d9b78d1272c6daf4a1dfb776ce66888821506805c8d7a20cf42c85144f0751b\": rpc error: code = NotFound desc = could not find container \"1d9b78d1272c6daf4a1dfb776ce66888821506805c8d7a20cf42c85144f0751b\": container with ID starting with 1d9b78d1272c6daf4a1dfb776ce66888821506805c8d7a20cf42c85144f0751b not found: ID does not exist"
Sep 11 22:17:23.641349 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.640389    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"]
Sep 11 22:17:23.641349 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:23.640850    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-95d6c2-openshift-default-64b748c86b-5qpnk"]
Sep 11 22:17:25.337096 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:25.337057    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fe4daef-73cf-4488-acec-a2b20a88db5e" path="/var/lib/kubelet/pods/9fe4daef-73cf-4488-acec-a2b20a88db5e/volumes"
Sep 11 22:17:27.417726 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:27.417699    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw6qf_7ab8a820-c9ec-415c-bfaa-3c9744ac040a/ovn-acl-logging/0.log"
Sep 11 22:17:27.425128 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:27.425105    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw6qf_7ab8a820-c9ec-415c-bfaa-3c9744ac040a/ovn-acl-logging/0.log"
Sep 11 22:17:27.623753 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:27.623719    2587 generic.go:356] "Generic (PLEG): container finished" podID="d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7" containerID="66e5cfa99aa777bd25ed932d9561c3226356e1df6b2254846497cba730cbbf0f" exitCode=0
Sep 11 22:17:27.623900 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:27.623793    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r" event={"ID":"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7","Type":"ContainerDied","Data":"66e5cfa99aa777bd25ed932d9561c3226356e1df6b2254846497cba730cbbf0f"}
Sep 11 22:17:27.624256 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:27.624241    2587 scope.go:117] "RemoveContainer" containerID="66e5cfa99aa777bd25ed932d9561c3226356e1df6b2254846497cba730cbbf0f"
Sep 11 22:17:27.625496 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:27.625467    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:17:27.625590 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:27.625534    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:17:27.625590 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:27.625567    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:17:28.636446 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:28.636406    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r" event={"ID":"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7","Type":"ContainerStarted","Data":"aaf8f277a56fab27bb6dc472e3bf60d14d86fd2d7dc911fbf92871f5ca8ba56f"}
Sep 11 22:17:29.265225 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:29.265180    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:17:29.266563 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:29.266535    2587 patch_prober.go:28] interesting pod/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 11 22:17:29.266695 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:29.266590    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r" podUID="d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 11 22:17:30.265835 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:30.265791    2587 patch_prober.go:28] interesting pod/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 11 22:17:30.266228 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:30.265854    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r" podUID="d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 11 22:17:31.265429 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:31.265390    2587 patch_prober.go:28] interesting pod/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 11 22:17:31.265619 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:31.265457    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r" podUID="d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 11 22:17:32.264906 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:32.264875    2587 patch_prober.go:28] interesting pod/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 11 22:17:32.265381 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:32.264927    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r" podUID="d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 11 22:17:33.265938 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:33.265897    2587 patch_prober.go:28] interesting pod/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 11 22:17:33.266315 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:33.265970    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r" podUID="d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 11 22:17:33.379422 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:33.379387    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:17:34.269836 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:34.269804    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:17:34.270817 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:34.270793    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:17:38.173718 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:38.173679    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4"]
Sep 11 22:17:38.174518 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:38.174477    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4" podUID="c66d78e8-8d56-4707-8de4-d7551dc143e8" containerName="maas-api" containerID="cri-o://92f719dcc94bafb4cd84e4c358d8228243eeefb1888b353748a97aeb0d2c532a" gracePeriod=30
Sep 11 22:17:38.224901 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:38.224855    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4" podUID="c66d78e8-8d56-4707-8de4-d7551dc143e8" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.74:8443/health\": dial tcp 10.134.0.74:8443: connect: connection refused"
Sep 11 22:17:38.446861 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:38.446835    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4"
Sep 11 22:17:38.525656 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:38.525618    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9vq4\" (UniqueName: \"kubernetes.io/projected/c66d78e8-8d56-4707-8de4-d7551dc143e8-kube-api-access-f9vq4\") pod \"c66d78e8-8d56-4707-8de4-d7551dc143e8\" (UID: \"c66d78e8-8d56-4707-8de4-d7551dc143e8\") "
Sep 11 22:17:38.525916 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:38.525897    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c66d78e8-8d56-4707-8de4-d7551dc143e8-openshift-service-ca\") pod \"c66d78e8-8d56-4707-8de4-d7551dc143e8\" (UID: \"c66d78e8-8d56-4707-8de4-d7551dc143e8\") "
Sep 11 22:17:38.526075 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:38.526060    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c66d78e8-8d56-4707-8de4-d7551dc143e8-maas-api-tls\") pod \"c66d78e8-8d56-4707-8de4-d7551dc143e8\" (UID: \"c66d78e8-8d56-4707-8de4-d7551dc143e8\") "
Sep 11 22:17:38.526181 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:38.526169    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c66d78e8-8d56-4707-8de4-d7551dc143e8-metrics-certs\") pod \"c66d78e8-8d56-4707-8de4-d7551dc143e8\" (UID: \"c66d78e8-8d56-4707-8de4-d7551dc143e8\") "
Sep 11 22:17:38.526439 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:38.526399    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c66d78e8-8d56-4707-8de4-d7551dc143e8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c66d78e8-8d56-4707-8de4-d7551dc143e8" (UID: "c66d78e8-8d56-4707-8de4-d7551dc143e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:17:38.526933 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:38.526909    2587 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c66d78e8-8d56-4707-8de4-d7551dc143e8-openshift-service-ca\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:38.528861 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:38.528818    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c66d78e8-8d56-4707-8de4-d7551dc143e8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c66d78e8-8d56-4707-8de4-d7551dc143e8" (UID: "c66d78e8-8d56-4707-8de4-d7551dc143e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:17:38.528984 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:38.528954    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c66d78e8-8d56-4707-8de4-d7551dc143e8-kube-api-access-f9vq4" (OuterVolumeSpecName: "kube-api-access-f9vq4") pod "c66d78e8-8d56-4707-8de4-d7551dc143e8" (UID: "c66d78e8-8d56-4707-8de4-d7551dc143e8"). InnerVolumeSpecName "kube-api-access-f9vq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:17:38.529095 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:38.529004    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c66d78e8-8d56-4707-8de4-d7551dc143e8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c66d78e8-8d56-4707-8de4-d7551dc143e8" (UID: "c66d78e8-8d56-4707-8de4-d7551dc143e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:17:38.628025 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:38.627983    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9vq4\" (UniqueName: \"kubernetes.io/projected/c66d78e8-8d56-4707-8de4-d7551dc143e8-kube-api-access-f9vq4\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:38.628025 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:38.628025    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c66d78e8-8d56-4707-8de4-d7551dc143e8-maas-api-tls\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:38.628233 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:38.628041    2587 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c66d78e8-8d56-4707-8de4-d7551dc143e8-metrics-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:38.699753 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:38.699652    2587 generic.go:356] "Generic (PLEG): container finished" podID="c66d78e8-8d56-4707-8de4-d7551dc143e8" containerID="92f719dcc94bafb4cd84e4c358d8228243eeefb1888b353748a97aeb0d2c532a" exitCode=0
Sep 11 22:17:38.699753 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:38.699696    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4"
Sep 11 22:17:38.699966 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:38.699747    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4" event={"ID":"c66d78e8-8d56-4707-8de4-d7551dc143e8","Type":"ContainerDied","Data":"92f719dcc94bafb4cd84e4c358d8228243eeefb1888b353748a97aeb0d2c532a"}
Sep 11 22:17:38.699966 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:38.699795    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4" event={"ID":"c66d78e8-8d56-4707-8de4-d7551dc143e8","Type":"ContainerDied","Data":"fe9913a7245e2073d4ce65ef7e22aadfb75ba8e1524663efe31a10c7730e0a15"}
Sep 11 22:17:38.699966 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:38.699818    2587 scope.go:117] "RemoveContainer" containerID="92f719dcc94bafb4cd84e4c358d8228243eeefb1888b353748a97aeb0d2c532a"
Sep 11 22:17:38.710078 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:38.710059    2587 scope.go:117] "RemoveContainer" containerID="92f719dcc94bafb4cd84e4c358d8228243eeefb1888b353748a97aeb0d2c532a"
Sep 11 22:17:38.710368 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:17:38.710350    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92f719dcc94bafb4cd84e4c358d8228243eeefb1888b353748a97aeb0d2c532a\": container with ID starting with 92f719dcc94bafb4cd84e4c358d8228243eeefb1888b353748a97aeb0d2c532a not found: ID does not exist" containerID="92f719dcc94bafb4cd84e4c358d8228243eeefb1888b353748a97aeb0d2c532a"
Sep 11 22:17:38.710435 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:38.710381    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f719dcc94bafb4cd84e4c358d8228243eeefb1888b353748a97aeb0d2c532a"} err="failed to get container status \"92f719dcc94bafb4cd84e4c358d8228243eeefb1888b353748a97aeb0d2c532a\": rpc error: code = NotFound desc = could not find container \"92f719dcc94bafb4cd84e4c358d8228243eeefb1888b353748a97aeb0d2c532a\": container with ID starting with 92f719dcc94bafb4cd84e4c358d8228243eeefb1888b353748a97aeb0d2c532a not found: ID does not exist"
Sep 11 22:17:38.725355 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:38.725306    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4"]
Sep 11 22:17:38.728298 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:38.728268    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fe6196-557c58f7db-6gzj4"]
Sep 11 22:17:39.337592 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:39.337556    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c66d78e8-8d56-4707-8de4-d7551dc143e8" path="/var/lib/kubelet/pods/c66d78e8-8d56-4707-8de4-d7551dc143e8/volumes"
Sep 11 22:17:44.847317 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:44.847281    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"]
Sep 11 22:17:44.847939 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:44.847878    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n" podUID="d42dd49d-9cc0-438a-a871-40e72a5e2e4e" containerName="istio-proxy" containerID="cri-o://77e0e6256ae1e19aa690ae7e9c2e1ec531437b526a1d333cd4f0dd8241b4a508" gracePeriod=30
Sep 11 22:17:47.870117 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:47.870077    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:17:50.109828 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.109767    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:17:50.245175 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.245142    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istio-token\") pod \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") "
Sep 11 22:17:50.245385 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.245188    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istio-data\") pod \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") "
Sep 11 22:17:50.245385 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.245209    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istio-envoy\") pod \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") "
Sep 11 22:17:50.245385 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.245252    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istiod-ca-cert\") pod \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") "
Sep 11 22:17:50.245572 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.245385    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-credential-socket\") pod \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") "
Sep 11 22:17:50.245572 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.245549    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istio-data" (OuterVolumeSpecName: "istio-data") pod "d42dd49d-9cc0-438a-a871-40e72a5e2e4e" (UID: "d42dd49d-9cc0-438a-a871-40e72a5e2e4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:50.245678 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.245631    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istio-podinfo\") pod \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") "
Sep 11 22:17:50.245678 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.245673    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-workload-certs\") pod \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") "
Sep 11 22:17:50.245790 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.245700    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znfs6\" (UniqueName: \"kubernetes.io/projected/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-kube-api-access-znfs6\") pod \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") "
Sep 11 22:17:50.245790 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.245732    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d42dd49d-9cc0-438a-a871-40e72a5e2e4e" (UID: "d42dd49d-9cc0-438a-a871-40e72a5e2e4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:17:50.245790 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.245746    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-workload-socket\") pod \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\" (UID: \"d42dd49d-9cc0-438a-a871-40e72a5e2e4e\") "
Sep 11 22:17:50.246039 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.245995    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d42dd49d-9cc0-438a-a871-40e72a5e2e4e" (UID: "d42dd49d-9cc0-438a-a871-40e72a5e2e4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:50.246156 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.246054    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d42dd49d-9cc0-438a-a871-40e72a5e2e4e" (UID: "d42dd49d-9cc0-438a-a871-40e72a5e2e4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:50.246249 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.246227    2587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istio-data\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:50.246455 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.246434    2587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istiod-ca-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:50.246563 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.246458    2587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-credential-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:50.246563 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.246472    2587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-workload-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:50.246563 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.246270    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d42dd49d-9cc0-438a-a871-40e72a5e2e4e" (UID: "d42dd49d-9cc0-438a-a871-40e72a5e2e4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:50.247748 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.247723    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d42dd49d-9cc0-438a-a871-40e72a5e2e4e" (UID: "d42dd49d-9cc0-438a-a871-40e72a5e2e4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:17:50.247954 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.247933    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istio-token" (OuterVolumeSpecName: "istio-token") pod "d42dd49d-9cc0-438a-a871-40e72a5e2e4e" (UID: "d42dd49d-9cc0-438a-a871-40e72a5e2e4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:17:50.248201 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.248165    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d42dd49d-9cc0-438a-a871-40e72a5e2e4e" (UID: "d42dd49d-9cc0-438a-a871-40e72a5e2e4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:17:50.248272 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.248195    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-kube-api-access-znfs6" (OuterVolumeSpecName: "kube-api-access-znfs6") pod "d42dd49d-9cc0-438a-a871-40e72a5e2e4e" (UID: "d42dd49d-9cc0-438a-a871-40e72a5e2e4e"). InnerVolumeSpecName "kube-api-access-znfs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:17:50.347637 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.347597    2587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istio-token\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:50.347637 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.347630    2587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istio-envoy\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:50.347637 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.347642    2587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-istio-podinfo\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:50.347880 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.347656    2587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-workload-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:50.347880 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.347667    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znfs6\" (UniqueName: \"kubernetes.io/projected/d42dd49d-9cc0-438a-a871-40e72a5e2e4e-kube-api-access-znfs6\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:17:50.777897 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.777858    2587 generic.go:356] "Generic (PLEG): container finished" podID="d42dd49d-9cc0-438a-a871-40e72a5e2e4e" containerID="77e0e6256ae1e19aa690ae7e9c2e1ec531437b526a1d333cd4f0dd8241b4a508" exitCode=0
Sep 11 22:17:50.778067 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.777908    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"
Sep 11 22:17:50.778067 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.777953    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n" event={"ID":"d42dd49d-9cc0-438a-a871-40e72a5e2e4e","Type":"ContainerDied","Data":"77e0e6256ae1e19aa690ae7e9c2e1ec531437b526a1d333cd4f0dd8241b4a508"}
Sep 11 22:17:50.778067 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.778009    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n" event={"ID":"d42dd49d-9cc0-438a-a871-40e72a5e2e4e","Type":"ContainerDied","Data":"85cd08db8c30b1e5a81e96ca87cdb4175cb36b59770e4a4f49c381ac8128b5e5"}
Sep 11 22:17:50.778067 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.778036    2587 scope.go:117] "RemoveContainer" containerID="77e0e6256ae1e19aa690ae7e9c2e1ec531437b526a1d333cd4f0dd8241b4a508"
Sep 11 22:17:50.788207 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.788184    2587 scope.go:117] "RemoveContainer" containerID="0e40e1ef489677ae8f1706d5516fc800d39fa052fe8f45fee91ed66fb780afc9"
Sep 11 22:17:50.797062 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.797040    2587 scope.go:117] "RemoveContainer" containerID="77e0e6256ae1e19aa690ae7e9c2e1ec531437b526a1d333cd4f0dd8241b4a508"
Sep 11 22:17:50.797370 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:17:50.797313    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77e0e6256ae1e19aa690ae7e9c2e1ec531437b526a1d333cd4f0dd8241b4a508\": container with ID starting with 77e0e6256ae1e19aa690ae7e9c2e1ec531437b526a1d333cd4f0dd8241b4a508 not found: ID does not exist" containerID="77e0e6256ae1e19aa690ae7e9c2e1ec531437b526a1d333cd4f0dd8241b4a508"
Sep 11 22:17:50.797370 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.797361    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77e0e6256ae1e19aa690ae7e9c2e1ec531437b526a1d333cd4f0dd8241b4a508"} err="failed to get container status \"77e0e6256ae1e19aa690ae7e9c2e1ec531437b526a1d333cd4f0dd8241b4a508\": rpc error: code = NotFound desc = could not find container \"77e0e6256ae1e19aa690ae7e9c2e1ec531437b526a1d333cd4f0dd8241b4a508\": container with ID starting with 77e0e6256ae1e19aa690ae7e9c2e1ec531437b526a1d333cd4f0dd8241b4a508 not found: ID does not exist"
Sep 11 22:17:50.797515 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.797378    2587 scope.go:117] "RemoveContainer" containerID="0e40e1ef489677ae8f1706d5516fc800d39fa052fe8f45fee91ed66fb780afc9"
Sep 11 22:17:50.797610 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:17:50.797595    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e40e1ef489677ae8f1706d5516fc800d39fa052fe8f45fee91ed66fb780afc9\": container with ID starting with 0e40e1ef489677ae8f1706d5516fc800d39fa052fe8f45fee91ed66fb780afc9 not found: ID does not exist" containerID="0e40e1ef489677ae8f1706d5516fc800d39fa052fe8f45fee91ed66fb780afc9"
Sep 11 22:17:50.797658 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.797612    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e40e1ef489677ae8f1706d5516fc800d39fa052fe8f45fee91ed66fb780afc9"} err="failed to get container status \"0e40e1ef489677ae8f1706d5516fc800d39fa052fe8f45fee91ed66fb780afc9\": rpc error: code = NotFound desc = could not find container \"0e40e1ef489677ae8f1706d5516fc800d39fa052fe8f45fee91ed66fb780afc9\": container with ID starting with 0e40e1ef489677ae8f1706d5516fc800d39fa052fe8f45fee91ed66fb780afc9 not found: ID does not exist"
Sep 11 22:17:50.802048 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.802028    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"]
Sep 11 22:17:50.806053 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:50.806033    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-fe6196-openshift-default-7f6b8b869d-qzs8n"]
Sep 11 22:17:51.336849 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:17:51.336817    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d42dd49d-9cc0-438a-a871-40e72a5e2e4e" path="/var/lib/kubelet/pods/d42dd49d-9cc0-438a-a871-40e72a5e2e4e/volumes"
Sep 11 22:18:05.962937 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:05.962902    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5"]
Sep 11 22:18:05.963469 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:05.963131    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5" podUID="42c7eecb-f974-4b7d-aea5-2190e35a35f4" containerName="maas-api" containerID="cri-o://4d66f8e1aa01b8f884a9263a2be884815ca92e43bb2cff00442dec667ffb3570" gracePeriod=30
Sep 11 22:18:06.216091 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:06.216033    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5"
Sep 11 22:18:06.314588 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:06.314547    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z59tj\" (UniqueName: \"kubernetes.io/projected/42c7eecb-f974-4b7d-aea5-2190e35a35f4-kube-api-access-z59tj\") pod \"42c7eecb-f974-4b7d-aea5-2190e35a35f4\" (UID: \"42c7eecb-f974-4b7d-aea5-2190e35a35f4\") "
Sep 11 22:18:06.314784 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:06.314603    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42c7eecb-f974-4b7d-aea5-2190e35a35f4-maas-api-tls\") pod \"42c7eecb-f974-4b7d-aea5-2190e35a35f4\" (UID: \"42c7eecb-f974-4b7d-aea5-2190e35a35f4\") "
Sep 11 22:18:06.314784 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:06.314633    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42c7eecb-f974-4b7d-aea5-2190e35a35f4-metrics-certs\") pod \"42c7eecb-f974-4b7d-aea5-2190e35a35f4\" (UID: \"42c7eecb-f974-4b7d-aea5-2190e35a35f4\") "
Sep 11 22:18:06.314784 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:06.314737    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c7eecb-f974-4b7d-aea5-2190e35a35f4-openshift-service-ca\") pod \"42c7eecb-f974-4b7d-aea5-2190e35a35f4\" (UID: \"42c7eecb-f974-4b7d-aea5-2190e35a35f4\") "
Sep 11 22:18:06.315130 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:06.315104    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42c7eecb-f974-4b7d-aea5-2190e35a35f4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "42c7eecb-f974-4b7d-aea5-2190e35a35f4" (UID: "42c7eecb-f974-4b7d-aea5-2190e35a35f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:18:06.316817 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:06.316787    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42c7eecb-f974-4b7d-aea5-2190e35a35f4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "42c7eecb-f974-4b7d-aea5-2190e35a35f4" (UID: "42c7eecb-f974-4b7d-aea5-2190e35a35f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:18:06.316913 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:06.316855    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c7eecb-f974-4b7d-aea5-2190e35a35f4-kube-api-access-z59tj" (OuterVolumeSpecName: "kube-api-access-z59tj") pod "42c7eecb-f974-4b7d-aea5-2190e35a35f4" (UID: "42c7eecb-f974-4b7d-aea5-2190e35a35f4"). InnerVolumeSpecName "kube-api-access-z59tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:06.316913 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:06.316857    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42c7eecb-f974-4b7d-aea5-2190e35a35f4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "42c7eecb-f974-4b7d-aea5-2190e35a35f4" (UID: "42c7eecb-f974-4b7d-aea5-2190e35a35f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:18:06.416497 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:06.416460    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z59tj\" (UniqueName: \"kubernetes.io/projected/42c7eecb-f974-4b7d-aea5-2190e35a35f4-kube-api-access-z59tj\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:06.416497 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:06.416497    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42c7eecb-f974-4b7d-aea5-2190e35a35f4-maas-api-tls\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:06.416705 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:06.416511    2587 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42c7eecb-f974-4b7d-aea5-2190e35a35f4-metrics-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:06.416705 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:06.416523    2587 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42c7eecb-f974-4b7d-aea5-2190e35a35f4-openshift-service-ca\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:06.871265 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:06.871232    2587 generic.go:356] "Generic (PLEG): container finished" podID="42c7eecb-f974-4b7d-aea5-2190e35a35f4" containerID="4d66f8e1aa01b8f884a9263a2be884815ca92e43bb2cff00442dec667ffb3570" exitCode=0
Sep 11 22:18:06.871481 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:06.871278    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5"
Sep 11 22:18:06.871481 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:06.871323    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5" event={"ID":"42c7eecb-f974-4b7d-aea5-2190e35a35f4","Type":"ContainerDied","Data":"4d66f8e1aa01b8f884a9263a2be884815ca92e43bb2cff00442dec667ffb3570"}
Sep 11 22:18:06.871481 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:06.871401    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5" event={"ID":"42c7eecb-f974-4b7d-aea5-2190e35a35f4","Type":"ContainerDied","Data":"8a6a04aa0c5b9f8f4518d477bb92cedb33908d7ce209f5f9e01179aca23b3eb1"}
Sep 11 22:18:06.871481 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:06.871428    2587 scope.go:117] "RemoveContainer" containerID="4d66f8e1aa01b8f884a9263a2be884815ca92e43bb2cff00442dec667ffb3570"
Sep 11 22:18:06.881456 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:06.881439    2587 scope.go:117] "RemoveContainer" containerID="4d66f8e1aa01b8f884a9263a2be884815ca92e43bb2cff00442dec667ffb3570"
Sep 11 22:18:06.881733 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:18:06.881715    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d66f8e1aa01b8f884a9263a2be884815ca92e43bb2cff00442dec667ffb3570\": container with ID starting with 4d66f8e1aa01b8f884a9263a2be884815ca92e43bb2cff00442dec667ffb3570 not found: ID does not exist" containerID="4d66f8e1aa01b8f884a9263a2be884815ca92e43bb2cff00442dec667ffb3570"
Sep 11 22:18:06.881781 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:06.881742    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d66f8e1aa01b8f884a9263a2be884815ca92e43bb2cff00442dec667ffb3570"} err="failed to get container status \"4d66f8e1aa01b8f884a9263a2be884815ca92e43bb2cff00442dec667ffb3570\": rpc error: code = NotFound desc = could not find container \"4d66f8e1aa01b8f884a9263a2be884815ca92e43bb2cff00442dec667ffb3570\": container with ID starting with 4d66f8e1aa01b8f884a9263a2be884815ca92e43bb2cff00442dec667ffb3570 not found: ID does not exist"
Sep 11 22:18:06.894037 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:06.894000    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5"]
Sep 11 22:18:06.898423 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:06.898402    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-7e2add-6568676f88-f82k5"]
Sep 11 22:18:07.336069 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:07.336036    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42c7eecb-f974-4b7d-aea5-2190e35a35f4" path="/var/lib/kubelet/pods/42c7eecb-f974-4b7d-aea5-2190e35a35f4/volumes"
Sep 11 22:18:09.270200 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:09.270157    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:18:11.225876 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:11.225832    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"]
Sep 11 22:18:11.226441 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:11.226110    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r" podUID="d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7" containerName="istio-proxy" containerID="cri-o://aaf8f277a56fab27bb6dc472e3bf60d14d86fd2d7dc911fbf92871f5ca8ba56f" gracePeriod=30
Sep 11 22:18:16.487127 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.487064    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:18:16.624385 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.624341    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istio-podinfo\") pod \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") "
Sep 11 22:18:16.624385 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.624394    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkcxw\" (UniqueName: \"kubernetes.io/projected/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-kube-api-access-rkcxw\") pod \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") "
Sep 11 22:18:16.624645 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.624456    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istio-envoy\") pod \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") "
Sep 11 22:18:16.624645 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.624519    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-workload-socket\") pod \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") "
Sep 11 22:18:16.624645 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.624548    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istio-data\") pod \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") "
Sep 11 22:18:16.624645 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.624572    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-workload-certs\") pod \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") "
Sep 11 22:18:16.624645 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.624593    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istio-token\") pod \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") "
Sep 11 22:18:16.624894 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.624670    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-credential-socket\") pod \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") "
Sep 11 22:18:16.624894 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.624739    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istiod-ca-cert\") pod \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\" (UID: \"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7\") "
Sep 11 22:18:16.624894 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.624875    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7" (UID: "d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:16.625054 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.624891    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istio-data" (OuterVolumeSpecName: "istio-data") pod "d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7" (UID: "d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:16.625054 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.624868    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7" (UID: "d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:16.625167 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.625118    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7" (UID: "d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:16.625221 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.625170    2587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-workload-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:16.625221 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.625191    2587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istio-data\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:16.625221 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.625205    2587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-workload-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:16.625221 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.625208    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7" (UID: "d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:18:16.626715 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.626691    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7" (UID: "d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:18:16.626805 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.626792    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istio-token" (OuterVolumeSpecName: "istio-token") pod "d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7" (UID: "d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:16.626991 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.626967    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7" (UID: "d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:16.626991 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.626973    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-kube-api-access-rkcxw" (OuterVolumeSpecName: "kube-api-access-rkcxw") pod "d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7" (UID: "d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7"). InnerVolumeSpecName "kube-api-access-rkcxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:16.726538 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.726505    2587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istio-token\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:16.726538 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.726533    2587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-credential-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:16.726538 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.726543    2587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istiod-ca-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:16.726815 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.726553    2587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istio-podinfo\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:16.726815 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.726562    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkcxw\" (UniqueName: \"kubernetes.io/projected/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-kube-api-access-rkcxw\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:16.726815 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.726570    2587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7-istio-envoy\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:16.932153 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.932117    2587 generic.go:356] "Generic (PLEG): container finished" podID="d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7" containerID="aaf8f277a56fab27bb6dc472e3bf60d14d86fd2d7dc911fbf92871f5ca8ba56f" exitCode=0
Sep 11 22:18:16.932373 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.932186    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"
Sep 11 22:18:16.932373 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.932187    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r" event={"ID":"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7","Type":"ContainerDied","Data":"aaf8f277a56fab27bb6dc472e3bf60d14d86fd2d7dc911fbf92871f5ca8ba56f"}
Sep 11 22:18:16.932513 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.932373    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r" event={"ID":"d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7","Type":"ContainerDied","Data":"edf72783fed5e18f5d8d471c1417017eb10c6e0ce16262f682e85e9372facb97"}
Sep 11 22:18:16.932513 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.932410    2587 scope.go:117] "RemoveContainer" containerID="aaf8f277a56fab27bb6dc472e3bf60d14d86fd2d7dc911fbf92871f5ca8ba56f"
Sep 11 22:18:16.942471 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.942453    2587 scope.go:117] "RemoveContainer" containerID="66e5cfa99aa777bd25ed932d9561c3226356e1df6b2254846497cba730cbbf0f"
Sep 11 22:18:16.950651 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.950628    2587 scope.go:117] "RemoveContainer" containerID="aaf8f277a56fab27bb6dc472e3bf60d14d86fd2d7dc911fbf92871f5ca8ba56f"
Sep 11 22:18:16.950926 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:18:16.950908    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaf8f277a56fab27bb6dc472e3bf60d14d86fd2d7dc911fbf92871f5ca8ba56f\": container with ID starting with aaf8f277a56fab27bb6dc472e3bf60d14d86fd2d7dc911fbf92871f5ca8ba56f not found: ID does not exist" containerID="aaf8f277a56fab27bb6dc472e3bf60d14d86fd2d7dc911fbf92871f5ca8ba56f"
Sep 11 22:18:16.950972 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.950936    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaf8f277a56fab27bb6dc472e3bf60d14d86fd2d7dc911fbf92871f5ca8ba56f"} err="failed to get container status \"aaf8f277a56fab27bb6dc472e3bf60d14d86fd2d7dc911fbf92871f5ca8ba56f\": rpc error: code = NotFound desc = could not find container \"aaf8f277a56fab27bb6dc472e3bf60d14d86fd2d7dc911fbf92871f5ca8ba56f\": container with ID starting with aaf8f277a56fab27bb6dc472e3bf60d14d86fd2d7dc911fbf92871f5ca8ba56f not found: ID does not exist"
Sep 11 22:18:16.950972 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.950954    2587 scope.go:117] "RemoveContainer" containerID="66e5cfa99aa777bd25ed932d9561c3226356e1df6b2254846497cba730cbbf0f"
Sep 11 22:18:16.951201 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:18:16.951183    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66e5cfa99aa777bd25ed932d9561c3226356e1df6b2254846497cba730cbbf0f\": container with ID starting with 66e5cfa99aa777bd25ed932d9561c3226356e1df6b2254846497cba730cbbf0f not found: ID does not exist" containerID="66e5cfa99aa777bd25ed932d9561c3226356e1df6b2254846497cba730cbbf0f"
Sep 11 22:18:16.951245 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.951206    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66e5cfa99aa777bd25ed932d9561c3226356e1df6b2254846497cba730cbbf0f"} err="failed to get container status \"66e5cfa99aa777bd25ed932d9561c3226356e1df6b2254846497cba730cbbf0f\": rpc error: code = NotFound desc = could not find container \"66e5cfa99aa777bd25ed932d9561c3226356e1df6b2254846497cba730cbbf0f\": container with ID starting with 66e5cfa99aa777bd25ed932d9561c3226356e1df6b2254846497cba730cbbf0f not found: ID does not exist"
Sep 11 22:18:16.955866 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.955843    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"]
Sep 11 22:18:16.959562 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:16.959542    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2715d7-openshift-default-656bd5d6cd-fzc6r"]
Sep 11 22:18:17.336469 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:17.336386    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7" path="/var/lib/kubelet/pods/d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7/volumes"
Sep 11 22:18:28.731515 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:28.731469    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"]
Sep 11 22:18:29.431023 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:29.430988    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"]
Sep 11 22:18:29.431278 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:29.431251    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk" podUID="70f777c7-66c9-4324-960d-7d8316053d1f" containerName="istio-proxy" containerID="cri-o://b9cc14caeb25bf621c23054689181f6f40f234fb307f31e6c71018ef333f4f66" gracePeriod=30
Sep 11 22:18:30.674518 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:30.674471    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:18:34.686115 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.686090    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:18:34.804816 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.804713    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-istio-data\") pod \"70f777c7-66c9-4324-960d-7d8316053d1f\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") "
Sep 11 22:18:34.804816 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.804770    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-credential-socket\") pod \"70f777c7-66c9-4324-960d-7d8316053d1f\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") "
Sep 11 22:18:34.804816 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.804792    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-istio-envoy\") pod \"70f777c7-66c9-4324-960d-7d8316053d1f\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") "
Sep 11 22:18:34.805101 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.804847    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-workload-socket\") pod \"70f777c7-66c9-4324-960d-7d8316053d1f\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") "
Sep 11 22:18:34.805101 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.804888    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f777c7-66c9-4324-960d-7d8316053d1f-istio-token\") pod \"70f777c7-66c9-4324-960d-7d8316053d1f\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") "
Sep 11 22:18:34.805101 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.804913    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-workload-certs\") pod \"70f777c7-66c9-4324-960d-7d8316053d1f\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") "
Sep 11 22:18:34.805101 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.804937    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f777c7-66c9-4324-960d-7d8316053d1f-istiod-ca-cert\") pod \"70f777c7-66c9-4324-960d-7d8316053d1f\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") "
Sep 11 22:18:34.805101 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.804974    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8djw\" (UniqueName: \"kubernetes.io/projected/70f777c7-66c9-4324-960d-7d8316053d1f-kube-api-access-l8djw\") pod \"70f777c7-66c9-4324-960d-7d8316053d1f\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") "
Sep 11 22:18:34.805101 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.805057    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f777c7-66c9-4324-960d-7d8316053d1f-istio-podinfo\") pod \"70f777c7-66c9-4324-960d-7d8316053d1f\" (UID: \"70f777c7-66c9-4324-960d-7d8316053d1f\") "
Sep 11 22:18:34.805467 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.805043    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-istio-data" (OuterVolumeSpecName: "istio-data") pod "70f777c7-66c9-4324-960d-7d8316053d1f" (UID: "70f777c7-66c9-4324-960d-7d8316053d1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:34.805467 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.805064    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "70f777c7-66c9-4324-960d-7d8316053d1f" (UID: "70f777c7-66c9-4324-960d-7d8316053d1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:34.805467 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.805206    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "70f777c7-66c9-4324-960d-7d8316053d1f" (UID: "70f777c7-66c9-4324-960d-7d8316053d1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:34.805467 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.805365    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "70f777c7-66c9-4324-960d-7d8316053d1f" (UID: "70f777c7-66c9-4324-960d-7d8316053d1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:34.805467 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.805454    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f777c7-66c9-4324-960d-7d8316053d1f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "70f777c7-66c9-4324-960d-7d8316053d1f" (UID: "70f777c7-66c9-4324-960d-7d8316053d1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:18:34.805724 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.805520    2587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-istio-data\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:34.805724 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.805535    2587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-credential-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:34.805724 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.805545    2587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-workload-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:34.805724 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.805553    2587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-workload-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:34.807244 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.807217    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f777c7-66c9-4324-960d-7d8316053d1f-kube-api-access-l8djw" (OuterVolumeSpecName: "kube-api-access-l8djw") pod "70f777c7-66c9-4324-960d-7d8316053d1f" (UID: "70f777c7-66c9-4324-960d-7d8316053d1f"). InnerVolumeSpecName "kube-api-access-l8djw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:34.807385 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.807299    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "70f777c7-66c9-4324-960d-7d8316053d1f" (UID: "70f777c7-66c9-4324-960d-7d8316053d1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:34.807608 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.807586    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f777c7-66c9-4324-960d-7d8316053d1f-istio-token" (OuterVolumeSpecName: "istio-token") pod "70f777c7-66c9-4324-960d-7d8316053d1f" (UID: "70f777c7-66c9-4324-960d-7d8316053d1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:34.807656 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.807610    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70f777c7-66c9-4324-960d-7d8316053d1f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "70f777c7-66c9-4324-960d-7d8316053d1f" (UID: "70f777c7-66c9-4324-960d-7d8316053d1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:18:34.906554 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.906523    2587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70f777c7-66c9-4324-960d-7d8316053d1f-istio-token\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:34.906554 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.906549    2587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70f777c7-66c9-4324-960d-7d8316053d1f-istiod-ca-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:34.906554 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.906558    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8djw\" (UniqueName: \"kubernetes.io/projected/70f777c7-66c9-4324-960d-7d8316053d1f-kube-api-access-l8djw\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:34.906790 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.906567    2587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70f777c7-66c9-4324-960d-7d8316053d1f-istio-podinfo\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:34.906790 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:34.906576    2587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70f777c7-66c9-4324-960d-7d8316053d1f-istio-envoy\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:35.032989 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:35.032957    2587 generic.go:356] "Generic (PLEG): container finished" podID="70f777c7-66c9-4324-960d-7d8316053d1f" containerID="b9cc14caeb25bf621c23054689181f6f40f234fb307f31e6c71018ef333f4f66" exitCode=0
Sep 11 22:18:35.033154 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:35.033015    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"
Sep 11 22:18:35.033154 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:35.033050    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk" event={"ID":"70f777c7-66c9-4324-960d-7d8316053d1f","Type":"ContainerDied","Data":"b9cc14caeb25bf621c23054689181f6f40f234fb307f31e6c71018ef333f4f66"}
Sep 11 22:18:35.033154 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:35.033111    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk" event={"ID":"70f777c7-66c9-4324-960d-7d8316053d1f","Type":"ContainerDied","Data":"abd2b169aedc25459df563e1d93eea14edcbe9c6fead4f21d969542a98d79bf9"}
Sep 11 22:18:35.033154 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:35.033142    2587 scope.go:117] "RemoveContainer" containerID="b9cc14caeb25bf621c23054689181f6f40f234fb307f31e6c71018ef333f4f66"
Sep 11 22:18:35.043240 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:35.043221    2587 scope.go:117] "RemoveContainer" containerID="41463a052eeed6d1ebd1bddf4f7eeb90934712f0cf77e84598ec95dc6cb0ec20"
Sep 11 22:18:35.051822 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:35.051803    2587 scope.go:117] "RemoveContainer" containerID="b9cc14caeb25bf621c23054689181f6f40f234fb307f31e6c71018ef333f4f66"
Sep 11 22:18:35.052110 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:18:35.052084    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9cc14caeb25bf621c23054689181f6f40f234fb307f31e6c71018ef333f4f66\": container with ID starting with b9cc14caeb25bf621c23054689181f6f40f234fb307f31e6c71018ef333f4f66 not found: ID does not exist" containerID="b9cc14caeb25bf621c23054689181f6f40f234fb307f31e6c71018ef333f4f66"
Sep 11 22:18:35.052211 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:35.052117    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9cc14caeb25bf621c23054689181f6f40f234fb307f31e6c71018ef333f4f66"} err="failed to get container status \"b9cc14caeb25bf621c23054689181f6f40f234fb307f31e6c71018ef333f4f66\": rpc error: code = NotFound desc = could not find container \"b9cc14caeb25bf621c23054689181f6f40f234fb307f31e6c71018ef333f4f66\": container with ID starting with b9cc14caeb25bf621c23054689181f6f40f234fb307f31e6c71018ef333f4f66 not found: ID does not exist"
Sep 11 22:18:35.052211 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:35.052135    2587 scope.go:117] "RemoveContainer" containerID="41463a052eeed6d1ebd1bddf4f7eeb90934712f0cf77e84598ec95dc6cb0ec20"
Sep 11 22:18:35.052399 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:18:35.052383    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41463a052eeed6d1ebd1bddf4f7eeb90934712f0cf77e84598ec95dc6cb0ec20\": container with ID starting with 41463a052eeed6d1ebd1bddf4f7eeb90934712f0cf77e84598ec95dc6cb0ec20 not found: ID does not exist" containerID="41463a052eeed6d1ebd1bddf4f7eeb90934712f0cf77e84598ec95dc6cb0ec20"
Sep 11 22:18:35.052448 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:35.052406    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41463a052eeed6d1ebd1bddf4f7eeb90934712f0cf77e84598ec95dc6cb0ec20"} err="failed to get container status \"41463a052eeed6d1ebd1bddf4f7eeb90934712f0cf77e84598ec95dc6cb0ec20\": rpc error: code = NotFound desc = could not find container \"41463a052eeed6d1ebd1bddf4f7eeb90934712f0cf77e84598ec95dc6cb0ec20\": container with ID starting with 41463a052eeed6d1ebd1bddf4f7eeb90934712f0cf77e84598ec95dc6cb0ec20 not found: ID does not exist"
Sep 11 22:18:35.055710 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:35.055653    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"]
Sep 11 22:18:35.059801 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:35.059780    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5ef59c-openshift-default-6d87659868-8jmhk"]
Sep 11 22:18:35.336439 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:35.336362    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70f777c7-66c9-4324-960d-7d8316053d1f" path="/var/lib/kubelet/pods/70f777c7-66c9-4324-960d-7d8316053d1f/volumes"
Sep 11 22:18:43.586038 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:43.585988    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:18:43.751118 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:43.751057    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46" podUID="7e795107-3b44-4fff-9d46-367022d34fc2" containerName="main" containerID="cri-o://73d37d8d199d31e9881c80d00154e7c2a24e471ffe2783cd5ef943cb14141d82" gracePeriod=45
Sep 11 22:18:44.015936 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.015902    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:18:44.087844 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.087806    2587 generic.go:356] "Generic (PLEG): container finished" podID="7e795107-3b44-4fff-9d46-367022d34fc2" containerID="73d37d8d199d31e9881c80d00154e7c2a24e471ffe2783cd5ef943cb14141d82" exitCode=0
Sep 11 22:18:44.088017 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.087855    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"
Sep 11 22:18:44.088017 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.087894    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46" event={"ID":"7e795107-3b44-4fff-9d46-367022d34fc2","Type":"ContainerDied","Data":"73d37d8d199d31e9881c80d00154e7c2a24e471ffe2783cd5ef943cb14141d82"}
Sep 11 22:18:44.088017 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.087942    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46" event={"ID":"7e795107-3b44-4fff-9d46-367022d34fc2","Type":"ContainerDied","Data":"b3a69fca5441473aafa7997f992dab4b53f03f99679272518cfad388bb798997"}
Sep 11 22:18:44.088017 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.087962    2587 scope.go:117] "RemoveContainer" containerID="73d37d8d199d31e9881c80d00154e7c2a24e471ffe2783cd5ef943cb14141d82"
Sep 11 22:18:44.088344 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.088314    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hh65x\" (UniqueName: \"kubernetes.io/projected/7e795107-3b44-4fff-9d46-367022d34fc2-kube-api-access-hh65x\") pod \"7e795107-3b44-4fff-9d46-367022d34fc2\" (UID: \"7e795107-3b44-4fff-9d46-367022d34fc2\") "
Sep 11 22:18:44.088397 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.088373    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e795107-3b44-4fff-9d46-367022d34fc2-home\") pod \"7e795107-3b44-4fff-9d46-367022d34fc2\" (UID: \"7e795107-3b44-4fff-9d46-367022d34fc2\") "
Sep 11 22:18:44.088448 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.088405    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e795107-3b44-4fff-9d46-367022d34fc2-tls-certs\") pod \"7e795107-3b44-4fff-9d46-367022d34fc2\" (UID: \"7e795107-3b44-4fff-9d46-367022d34fc2\") "
Sep 11 22:18:44.088448 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.088422    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e795107-3b44-4fff-9d46-367022d34fc2-model-cache\") pod \"7e795107-3b44-4fff-9d46-367022d34fc2\" (UID: \"7e795107-3b44-4fff-9d46-367022d34fc2\") "
Sep 11 22:18:44.088562 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.088465    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e795107-3b44-4fff-9d46-367022d34fc2-tmp-dir\") pod \"7e795107-3b44-4fff-9d46-367022d34fc2\" (UID: \"7e795107-3b44-4fff-9d46-367022d34fc2\") "
Sep 11 22:18:44.088617 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.088591    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e795107-3b44-4fff-9d46-367022d34fc2-dshm\") pod \"7e795107-3b44-4fff-9d46-367022d34fc2\" (UID: \"7e795107-3b44-4fff-9d46-367022d34fc2\") "
Sep 11 22:18:44.088723 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.088694    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e795107-3b44-4fff-9d46-367022d34fc2-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e795107-3b44-4fff-9d46-367022d34fc2" (UID: "7e795107-3b44-4fff-9d46-367022d34fc2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:44.088921 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.088891    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e795107-3b44-4fff-9d46-367022d34fc2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e795107-3b44-4fff-9d46-367022d34fc2" (UID: "7e795107-3b44-4fff-9d46-367022d34fc2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:44.088921 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.088701    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e795107-3b44-4fff-9d46-367022d34fc2-home" (OuterVolumeSpecName: "home") pod "7e795107-3b44-4fff-9d46-367022d34fc2" (UID: "7e795107-3b44-4fff-9d46-367022d34fc2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:44.089180 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.088942    2587 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e795107-3b44-4fff-9d46-367022d34fc2-model-cache\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:44.090861 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.090824    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e795107-3b44-4fff-9d46-367022d34fc2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e795107-3b44-4fff-9d46-367022d34fc2" (UID: "7e795107-3b44-4fff-9d46-367022d34fc2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:18:44.090861 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.090839    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e795107-3b44-4fff-9d46-367022d34fc2-kube-api-access-hh65x" (OuterVolumeSpecName: "kube-api-access-hh65x") pod "7e795107-3b44-4fff-9d46-367022d34fc2" (UID: "7e795107-3b44-4fff-9d46-367022d34fc2"). InnerVolumeSpecName "kube-api-access-hh65x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:44.091025 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.090829    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e795107-3b44-4fff-9d46-367022d34fc2-dshm" (OuterVolumeSpecName: "dshm") pod "7e795107-3b44-4fff-9d46-367022d34fc2" (UID: "7e795107-3b44-4fff-9d46-367022d34fc2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:44.111008 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.110985    2587 scope.go:117] "RemoveContainer" containerID="73d37d8d199d31e9881c80d00154e7c2a24e471ffe2783cd5ef943cb14141d82"
Sep 11 22:18:44.111388 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:18:44.111366    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d37d8d199d31e9881c80d00154e7c2a24e471ffe2783cd5ef943cb14141d82\": container with ID starting with 73d37d8d199d31e9881c80d00154e7c2a24e471ffe2783cd5ef943cb14141d82 not found: ID does not exist" containerID="73d37d8d199d31e9881c80d00154e7c2a24e471ffe2783cd5ef943cb14141d82"
Sep 11 22:18:44.111449 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.111398    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d37d8d199d31e9881c80d00154e7c2a24e471ffe2783cd5ef943cb14141d82"} err="failed to get container status \"73d37d8d199d31e9881c80d00154e7c2a24e471ffe2783cd5ef943cb14141d82\": rpc error: code = NotFound desc = could not find container \"73d37d8d199d31e9881c80d00154e7c2a24e471ffe2783cd5ef943cb14141d82\": container with ID starting with 73d37d8d199d31e9881c80d00154e7c2a24e471ffe2783cd5ef943cb14141d82 not found: ID does not exist"
Sep 11 22:18:44.190531 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.190491    2587 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e795107-3b44-4fff-9d46-367022d34fc2-dshm\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:44.190531 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.190529    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh65x\" (UniqueName: \"kubernetes.io/projected/7e795107-3b44-4fff-9d46-367022d34fc2-kube-api-access-hh65x\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:44.190754 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.190539    2587 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e795107-3b44-4fff-9d46-367022d34fc2-home\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:44.190754 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.190551    2587 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e795107-3b44-4fff-9d46-367022d34fc2-tls-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:44.190754 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.190561    2587 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e795107-3b44-4fff-9d46-367022d34fc2-tmp-dir\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:44.412588 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.412548    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"]
Sep 11 22:18:44.416617 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:44.416586    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d09c96/auth-test-model-d09c96-kserve-776898dbdc-qbg46"]
Sep 11 22:18:45.336438 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:45.336404    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e795107-3b44-4fff-9d46-367022d34fc2" path="/var/lib/kubelet/pods/7e795107-3b44-4fff-9d46-367022d34fc2/volumes"
Sep 11 22:18:47.048735 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:47.048699    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"]
Sep 11 22:18:47.049197 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:47.048945    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2" podUID="07be57ee-7747-4608-820d-b20afeaf3bd9" containerName="istio-proxy" containerID="cri-o://009726f2597f48ac17ed6b324b4853cf2dbd5c66d6e594c7d1fa1f7686675130" gracePeriod=30
Sep 11 22:18:52.304864 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.304840    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:18:52.373288 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.373215    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07be57ee-7747-4608-820d-b20afeaf3bd9-istio-podinfo\") pod \"07be57ee-7747-4608-820d-b20afeaf3bd9\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") "
Sep 11 22:18:52.373288 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.373256    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9c6v\" (UniqueName: \"kubernetes.io/projected/07be57ee-7747-4608-820d-b20afeaf3bd9-kube-api-access-n9c6v\") pod \"07be57ee-7747-4608-820d-b20afeaf3bd9\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") "
Sep 11 22:18:52.373288 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.373286    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07be57ee-7747-4608-820d-b20afeaf3bd9-istiod-ca-cert\") pod \"07be57ee-7747-4608-820d-b20afeaf3bd9\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") "
Sep 11 22:18:52.373591 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.373367    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-credential-socket\") pod \"07be57ee-7747-4608-820d-b20afeaf3bd9\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") "
Sep 11 22:18:52.373591 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.373400    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07be57ee-7747-4608-820d-b20afeaf3bd9-istio-token\") pod \"07be57ee-7747-4608-820d-b20afeaf3bd9\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") "
Sep 11 22:18:52.373591 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.373442    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-istio-envoy\") pod \"07be57ee-7747-4608-820d-b20afeaf3bd9\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") "
Sep 11 22:18:52.373591 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.373460    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-workload-socket\") pod \"07be57ee-7747-4608-820d-b20afeaf3bd9\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") "
Sep 11 22:18:52.373591 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.373489    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-istio-data\") pod \"07be57ee-7747-4608-820d-b20afeaf3bd9\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") "
Sep 11 22:18:52.373591 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.373526    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-workload-certs\") pod \"07be57ee-7747-4608-820d-b20afeaf3bd9\" (UID: \"07be57ee-7747-4608-820d-b20afeaf3bd9\") "
Sep 11 22:18:52.373896 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.373734    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "07be57ee-7747-4608-820d-b20afeaf3bd9" (UID: "07be57ee-7747-4608-820d-b20afeaf3bd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:52.373896 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.373758    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07be57ee-7747-4608-820d-b20afeaf3bd9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "07be57ee-7747-4608-820d-b20afeaf3bd9" (UID: "07be57ee-7747-4608-820d-b20afeaf3bd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:18:52.373896 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.373852    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "07be57ee-7747-4608-820d-b20afeaf3bd9" (UID: "07be57ee-7747-4608-820d-b20afeaf3bd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:52.374058 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.373947    2587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-workload-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:52.374058 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.373956    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "07be57ee-7747-4608-820d-b20afeaf3bd9" (UID: "07be57ee-7747-4608-820d-b20afeaf3bd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:52.374058 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.373971    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-istio-data" (OuterVolumeSpecName: "istio-data") pod "07be57ee-7747-4608-820d-b20afeaf3bd9" (UID: "07be57ee-7747-4608-820d-b20afeaf3bd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:52.374058 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.373966    2587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07be57ee-7747-4608-820d-b20afeaf3bd9-istiod-ca-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:52.374058 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.374008    2587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-credential-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:52.375637 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.375606    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07be57ee-7747-4608-820d-b20afeaf3bd9-kube-api-access-n9c6v" (OuterVolumeSpecName: "kube-api-access-n9c6v") pod "07be57ee-7747-4608-820d-b20afeaf3bd9" (UID: "07be57ee-7747-4608-820d-b20afeaf3bd9"). InnerVolumeSpecName "kube-api-access-n9c6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:52.375744 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.375636    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07be57ee-7747-4608-820d-b20afeaf3bd9-istio-token" (OuterVolumeSpecName: "istio-token") pod "07be57ee-7747-4608-820d-b20afeaf3bd9" (UID: "07be57ee-7747-4608-820d-b20afeaf3bd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:18:52.375744 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.375641    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "07be57ee-7747-4608-820d-b20afeaf3bd9" (UID: "07be57ee-7747-4608-820d-b20afeaf3bd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:18:52.375744 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.375672    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07be57ee-7747-4608-820d-b20afeaf3bd9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "07be57ee-7747-4608-820d-b20afeaf3bd9" (UID: "07be57ee-7747-4608-820d-b20afeaf3bd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:18:52.474998 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.474961    2587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07be57ee-7747-4608-820d-b20afeaf3bd9-istio-token\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:52.474998 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.474991    2587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-istio-envoy\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:52.474998 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.474999    2587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-istio-data\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:52.474998 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.475007    2587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07be57ee-7747-4608-820d-b20afeaf3bd9-workload-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:52.475266 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.475017    2587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07be57ee-7747-4608-820d-b20afeaf3bd9-istio-podinfo\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:52.475266 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:52.475030    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9c6v\" (UniqueName: \"kubernetes.io/projected/07be57ee-7747-4608-820d-b20afeaf3bd9-kube-api-access-n9c6v\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:18:53.138512 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:53.138478    2587 generic.go:356] "Generic (PLEG): container finished" podID="07be57ee-7747-4608-820d-b20afeaf3bd9" containerID="009726f2597f48ac17ed6b324b4853cf2dbd5c66d6e594c7d1fa1f7686675130" exitCode=0
Sep 11 22:18:53.138724 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:53.138536    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"
Sep 11 22:18:53.138724 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:53.138570    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2" event={"ID":"07be57ee-7747-4608-820d-b20afeaf3bd9","Type":"ContainerDied","Data":"009726f2597f48ac17ed6b324b4853cf2dbd5c66d6e594c7d1fa1f7686675130"}
Sep 11 22:18:53.138724 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:53.138628    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2" event={"ID":"07be57ee-7747-4608-820d-b20afeaf3bd9","Type":"ContainerDied","Data":"76eac1399a749e4ca8aca90162da935c44b588796522877a17774e240a3bd276"}
Sep 11 22:18:53.138724 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:53.138653    2587 scope.go:117] "RemoveContainer" containerID="009726f2597f48ac17ed6b324b4853cf2dbd5c66d6e594c7d1fa1f7686675130"
Sep 11 22:18:53.148433 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:53.148400    2587 scope.go:117] "RemoveContainer" containerID="d98cd1e47b35733ce05bff04f95285987c5ce2184ddd8b736e97353fd30bed2b"
Sep 11 22:18:53.156761 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:53.156743    2587 scope.go:117] "RemoveContainer" containerID="009726f2597f48ac17ed6b324b4853cf2dbd5c66d6e594c7d1fa1f7686675130"
Sep 11 22:18:53.157310 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:18:53.157290    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"009726f2597f48ac17ed6b324b4853cf2dbd5c66d6e594c7d1fa1f7686675130\": container with ID starting with 009726f2597f48ac17ed6b324b4853cf2dbd5c66d6e594c7d1fa1f7686675130 not found: ID does not exist" containerID="009726f2597f48ac17ed6b324b4853cf2dbd5c66d6e594c7d1fa1f7686675130"
Sep 11 22:18:53.157393 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:53.157318    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"009726f2597f48ac17ed6b324b4853cf2dbd5c66d6e594c7d1fa1f7686675130"} err="failed to get container status \"009726f2597f48ac17ed6b324b4853cf2dbd5c66d6e594c7d1fa1f7686675130\": rpc error: code = NotFound desc = could not find container \"009726f2597f48ac17ed6b324b4853cf2dbd5c66d6e594c7d1fa1f7686675130\": container with ID starting with 009726f2597f48ac17ed6b324b4853cf2dbd5c66d6e594c7d1fa1f7686675130 not found: ID does not exist"
Sep 11 22:18:53.157393 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:53.157351    2587 scope.go:117] "RemoveContainer" containerID="d98cd1e47b35733ce05bff04f95285987c5ce2184ddd8b736e97353fd30bed2b"
Sep 11 22:18:53.157574 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:18:53.157561    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d98cd1e47b35733ce05bff04f95285987c5ce2184ddd8b736e97353fd30bed2b\": container with ID starting with d98cd1e47b35733ce05bff04f95285987c5ce2184ddd8b736e97353fd30bed2b not found: ID does not exist" containerID="d98cd1e47b35733ce05bff04f95285987c5ce2184ddd8b736e97353fd30bed2b"
Sep 11 22:18:53.157619 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:53.157577    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d98cd1e47b35733ce05bff04f95285987c5ce2184ddd8b736e97353fd30bed2b"} err="failed to get container status \"d98cd1e47b35733ce05bff04f95285987c5ce2184ddd8b736e97353fd30bed2b\": rpc error: code = NotFound desc = could not find container \"d98cd1e47b35733ce05bff04f95285987c5ce2184ddd8b736e97353fd30bed2b\": container with ID starting with d98cd1e47b35733ce05bff04f95285987c5ce2184ddd8b736e97353fd30bed2b not found: ID does not exist"
Sep 11 22:18:53.160391 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:53.160369    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"]
Sep 11 22:18:53.165471 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:53.165452    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-38f772-openshift-default-66f66df8c8-kclk2"]
Sep 11 22:18:53.336678 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:18:53.336645    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07be57ee-7747-4608-820d-b20afeaf3bd9" path="/var/lib/kubelet/pods/07be57ee-7747-4608-820d-b20afeaf3bd9/volumes"
Sep 11 22:19:20.081616 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:20.081528    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:19:38.372264 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:38.372225    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:19:50.972553 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:50.972513    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:19:53.214865 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.214827    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7684-0701c6"]
Sep 11 22:19:53.215298 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215282    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fe4daef-73cf-4488-acec-a2b20a88db5e" containerName="istio-proxy"
Sep 11 22:19:53.215368 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215299    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fe4daef-73cf-4488-acec-a2b20a88db5e" containerName="istio-proxy"
Sep 11 22:19:53.215368 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215309    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7" containerName="istio-proxy"
Sep 11 22:19:53.215368 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215315    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7" containerName="istio-proxy"
Sep 11 22:19:53.215368 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215347    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c66d78e8-8d56-4707-8de4-d7551dc143e8" containerName="maas-api"
Sep 11 22:19:53.215368 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215357    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="c66d78e8-8d56-4707-8de4-d7551dc143e8" containerName="maas-api"
Sep 11 22:19:53.215533 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215371    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07be57ee-7747-4608-820d-b20afeaf3bd9" containerName="istio-proxy"
Sep 11 22:19:53.215533 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215376    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="07be57ee-7747-4608-820d-b20afeaf3bd9" containerName="istio-proxy"
Sep 11 22:19:53.215533 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215384    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d42dd49d-9cc0-438a-a871-40e72a5e2e4e" containerName="istio-proxy"
Sep 11 22:19:53.215533 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215389    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="d42dd49d-9cc0-438a-a871-40e72a5e2e4e" containerName="istio-proxy"
Sep 11 22:19:53.215533 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215397    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7" containerName="istio-proxy"
Sep 11 22:19:53.215533 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215402    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7" containerName="istio-proxy"
Sep 11 22:19:53.215533 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215410    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07be57ee-7747-4608-820d-b20afeaf3bd9" containerName="istio-proxy"
Sep 11 22:19:53.215533 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215415    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="07be57ee-7747-4608-820d-b20afeaf3bd9" containerName="istio-proxy"
Sep 11 22:19:53.215533 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215425    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42c7eecb-f974-4b7d-aea5-2190e35a35f4" containerName="maas-api"
Sep 11 22:19:53.215533 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215434    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="42c7eecb-f974-4b7d-aea5-2190e35a35f4" containerName="maas-api"
Sep 11 22:19:53.215533 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215440    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54e64fba-6ece-4dee-8e5f-76210f2c4ed8" containerName="maas-api"
Sep 11 22:19:53.215533 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215445    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="54e64fba-6ece-4dee-8e5f-76210f2c4ed8" containerName="maas-api"
Sep 11 22:19:53.215533 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215453    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70f777c7-66c9-4324-960d-7d8316053d1f" containerName="istio-proxy"
Sep 11 22:19:53.215533 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215459    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="70f777c7-66c9-4324-960d-7d8316053d1f" containerName="istio-proxy"
Sep 11 22:19:53.215533 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215465    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70f777c7-66c9-4324-960d-7d8316053d1f" containerName="istio-proxy"
Sep 11 22:19:53.215533 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215470    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="70f777c7-66c9-4324-960d-7d8316053d1f" containerName="istio-proxy"
Sep 11 22:19:53.215533 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215476    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e795107-3b44-4fff-9d46-367022d34fc2" containerName="main"
Sep 11 22:19:53.215533 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215481    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e795107-3b44-4fff-9d46-367022d34fc2" containerName="main"
Sep 11 22:19:53.215533 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215486    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d42dd49d-9cc0-438a-a871-40e72a5e2e4e" containerName="istio-proxy"
Sep 11 22:19:53.215533 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215491    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="d42dd49d-9cc0-438a-a871-40e72a5e2e4e" containerName="istio-proxy"
Sep 11 22:19:53.216131 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215581    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="d42dd49d-9cc0-438a-a871-40e72a5e2e4e" containerName="istio-proxy"
Sep 11 22:19:53.216131 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215594    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="70f777c7-66c9-4324-960d-7d8316053d1f" containerName="istio-proxy"
Sep 11 22:19:53.216131 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215600    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="07be57ee-7747-4608-820d-b20afeaf3bd9" containerName="istio-proxy"
Sep 11 22:19:53.216131 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215607    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7" containerName="istio-proxy"
Sep 11 22:19:53.216131 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215613    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="07be57ee-7747-4608-820d-b20afeaf3bd9" containerName="istio-proxy"
Sep 11 22:19:53.216131 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215619    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fe4daef-73cf-4488-acec-a2b20a88db5e" containerName="istio-proxy"
Sep 11 22:19:53.216131 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215624    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8a830dd-b1fe-4c3a-8eb7-c102b2fa05e7" containerName="istio-proxy"
Sep 11 22:19:53.216131 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215629    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="42c7eecb-f974-4b7d-aea5-2190e35a35f4" containerName="maas-api"
Sep 11 22:19:53.216131 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215635    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="54e64fba-6ece-4dee-8e5f-76210f2c4ed8" containerName="maas-api"
Sep 11 22:19:53.216131 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215640    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="70f777c7-66c9-4324-960d-7d8316053d1f" containerName="istio-proxy"
Sep 11 22:19:53.216131 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215646    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="c66d78e8-8d56-4707-8de4-d7551dc143e8" containerName="maas-api"
Sep 11 22:19:53.216131 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.215652    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e795107-3b44-4fff-9d46-367022d34fc2" containerName="main"
Sep 11 22:19:53.218965 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.218948    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-0701c6"
Sep 11 22:19:53.222100 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.222076    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h4c6b\""
Sep 11 22:19:53.224570 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.224544    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-0701c6"]
Sep 11 22:19:53.366388 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.366312    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svfv2\" (UniqueName: \"kubernetes.io/projected/d5a61eb0-8aa4-4004-a6ee-ed18256145e0-kube-api-access-svfv2\") pod \"test-curl-7684-0701c6\" (UID: \"d5a61eb0-8aa4-4004-a6ee-ed18256145e0\") " pod="opendatahub/test-curl-7684-0701c6"
Sep 11 22:19:53.467443 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.467354    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svfv2\" (UniqueName: \"kubernetes.io/projected/d5a61eb0-8aa4-4004-a6ee-ed18256145e0-kube-api-access-svfv2\") pod \"test-curl-7684-0701c6\" (UID: \"d5a61eb0-8aa4-4004-a6ee-ed18256145e0\") " pod="opendatahub/test-curl-7684-0701c6"
Sep 11 22:19:53.475120 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.475096    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svfv2\" (UniqueName: \"kubernetes.io/projected/d5a61eb0-8aa4-4004-a6ee-ed18256145e0-kube-api-access-svfv2\") pod \"test-curl-7684-0701c6\" (UID: \"d5a61eb0-8aa4-4004-a6ee-ed18256145e0\") " pod="opendatahub/test-curl-7684-0701c6"
Sep 11 22:19:53.530814 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.530777    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-0701c6"
Sep 11 22:19:53.807776 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:53.807751    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-0701c6"]
Sep 11 22:19:53.809454 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:19:53.809425    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5a61eb0_8aa4_4004_a6ee_ed18256145e0.slice/crio-5bf4bf710da836130fcce9bd429fec0c80b8062be2f37d40633cfa3a56b1814c WatchSource:0}: Error finding container 5bf4bf710da836130fcce9bd429fec0c80b8062be2f37d40633cfa3a56b1814c: Status 404 returned error can't find the container with id 5bf4bf710da836130fcce9bd429fec0c80b8062be2f37d40633cfa3a56b1814c
Sep 11 22:19:54.452120 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:54.452083    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-0701c6" event={"ID":"d5a61eb0-8aa4-4004-a6ee-ed18256145e0","Type":"ContainerStarted","Data":"f9aae88ec610906324e88fdcb4ca8a823ee04faa8fb0a9287d199ce608b9d346"}
Sep 11 22:19:54.452120 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:54.452123    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-0701c6" event={"ID":"d5a61eb0-8aa4-4004-a6ee-ed18256145e0","Type":"ContainerStarted","Data":"5bf4bf710da836130fcce9bd429fec0c80b8062be2f37d40633cfa3a56b1814c"}
Sep 11 22:19:54.468716 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:54.468664    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7684-0701c6" podStartSLOduration=1.468648232 podStartE2EDuration="1.468648232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:19:54.46722231 +0000 UTC m=+1647.642298681" watchObservedRunningTime="2026-09-11 22:19:54.468648232 +0000 UTC m=+1647.643724603"
Sep 11 22:19:55.069455 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:55.069417    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7684-0701c6"]
Sep 11 22:19:55.072381 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:55.072347    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7684-0701c6"]
Sep 11 22:19:55.196313 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:55.196275    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7684-dbd053"]
Sep 11 22:19:55.196684 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:55.196670    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a61eb0-8aa4-4004-a6ee-ed18256145e0" containerName="test-curl-7684-0701c6"
Sep 11 22:19:55.196726 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:55.196686    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a61eb0-8aa4-4004-a6ee-ed18256145e0" containerName="test-curl-7684-0701c6"
Sep 11 22:19:55.196726 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:55.196710    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fe4daef-73cf-4488-acec-a2b20a88db5e" containerName="istio-proxy"
Sep 11 22:19:55.196726 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:55.196715    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fe4daef-73cf-4488-acec-a2b20a88db5e" containerName="istio-proxy"
Sep 11 22:19:55.196825 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:55.196777    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fe4daef-73cf-4488-acec-a2b20a88db5e" containerName="istio-proxy"
Sep 11 22:19:55.196825 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:55.196786    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5a61eb0-8aa4-4004-a6ee-ed18256145e0" containerName="test-curl-7684-0701c6"
Sep 11 22:19:55.196825 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:55.196794    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="d42dd49d-9cc0-438a-a871-40e72a5e2e4e" containerName="istio-proxy"
Sep 11 22:19:55.199951 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:55.199935    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-dbd053"
Sep 11 22:19:55.205295 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:55.205269    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-dbd053"]
Sep 11 22:19:55.387194 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:55.387162    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5npdd\" (UniqueName: \"kubernetes.io/projected/0d5e42bd-a325-4c11-95f0-857525e4697c-kube-api-access-5npdd\") pod \"test-curl-7684-dbd053\" (UID: \"0d5e42bd-a325-4c11-95f0-857525e4697c\") " pod="opendatahub/test-curl-7684-dbd053"
Sep 11 22:19:55.457910 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:19:55.457832    2587 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7684-0701c6" container="test-curl-7684-0701c6"
Sep 11 22:19:55.488189 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:55.488157    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5npdd\" (UniqueName: \"kubernetes.io/projected/0d5e42bd-a325-4c11-95f0-857525e4697c-kube-api-access-5npdd\") pod \"test-curl-7684-dbd053\" (UID: \"0d5e42bd-a325-4c11-95f0-857525e4697c\") " pod="opendatahub/test-curl-7684-dbd053"
Sep 11 22:19:55.496624 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:55.496600    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5npdd\" (UniqueName: \"kubernetes.io/projected/0d5e42bd-a325-4c11-95f0-857525e4697c-kube-api-access-5npdd\") pod \"test-curl-7684-dbd053\" (UID: \"0d5e42bd-a325-4c11-95f0-857525e4697c\") " pod="opendatahub/test-curl-7684-dbd053"
Sep 11 22:19:55.511488 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:55.511460    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-dbd053"
Sep 11 22:19:55.784940 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:55.784910    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7684-dbd053"]
Sep 11 22:19:55.786228 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:19:55.786188    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d5e42bd_a325_4c11_95f0_857525e4697c.slice/crio-357ceaf5831520e3e97a638c0e9f9f53e5456a2fb536ef19d681b52a9e3f969e WatchSource:0}: Error finding container 357ceaf5831520e3e97a638c0e9f9f53e5456a2fb536ef19d681b52a9e3f969e: Status 404 returned error can't find the container with id 357ceaf5831520e3e97a638c0e9f9f53e5456a2fb536ef19d681b52a9e3f969e
Sep 11 22:19:56.464634 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:56.464595    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-dbd053" event={"ID":"0d5e42bd-a325-4c11-95f0-857525e4697c","Type":"ContainerStarted","Data":"9a5631c5a15a046eedcf0290ebf2cea5d329e56b0cfaba59de408495b86c5b58"}
Sep 11 22:19:56.464634 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:56.464638    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7684-dbd053" event={"ID":"0d5e42bd-a325-4c11-95f0-857525e4697c","Type":"ContainerStarted","Data":"357ceaf5831520e3e97a638c0e9f9f53e5456a2fb536ef19d681b52a9e3f969e"}
Sep 11 22:19:56.465118 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:56.464692    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7684-0701c6" podUID="d5a61eb0-8aa4-4004-a6ee-ed18256145e0" containerName="test-curl-7684-0701c6" containerID="cri-o://f9aae88ec610906324e88fdcb4ca8a823ee04faa8fb0a9287d199ce608b9d346" gracePeriod=2
Sep 11 22:19:56.480199 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:56.480150    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7684-dbd053" podStartSLOduration=1.480137191 podStartE2EDuration="1.480137191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:19:56.479291228 +0000 UTC m=+1649.654367598" watchObservedRunningTime="2026-09-11 22:19:56.480137191 +0000 UTC m=+1649.655213560"
Sep 11 22:19:57.073768 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:57.073727    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7684-dbd053"]
Sep 11 22:19:57.079344 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:57.079304    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7684-dbd053"]
Sep 11 22:19:57.469642 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:19:57.469612    2587 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7684-dbd053" container="test-curl-7684-dbd053"
Sep 11 22:19:58.476071 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:58.476013    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7684-dbd053" podUID="0d5e42bd-a325-4c11-95f0-857525e4697c" containerName="test-curl-7684-dbd053" containerID="cri-o://9a5631c5a15a046eedcf0290ebf2cea5d329e56b0cfaba59de408495b86c5b58" gracePeriod=2
Sep 11 22:19:58.615085 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:58.615062    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-0701c6"
Sep 11 22:19:58.713321 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:58.713281    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svfv2\" (UniqueName: \"kubernetes.io/projected/d5a61eb0-8aa4-4004-a6ee-ed18256145e0-kube-api-access-svfv2\") pod \"d5a61eb0-8aa4-4004-a6ee-ed18256145e0\" (UID: \"d5a61eb0-8aa4-4004-a6ee-ed18256145e0\") "
Sep 11 22:19:58.715364 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:58.715338    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a61eb0-8aa4-4004-a6ee-ed18256145e0-kube-api-access-svfv2" (OuterVolumeSpecName: "kube-api-access-svfv2") pod "d5a61eb0-8aa4-4004-a6ee-ed18256145e0" (UID: "d5a61eb0-8aa4-4004-a6ee-ed18256145e0"). InnerVolumeSpecName "kube-api-access-svfv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:19:58.814714 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:58.814674    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svfv2\" (UniqueName: \"kubernetes.io/projected/d5a61eb0-8aa4-4004-a6ee-ed18256145e0-kube-api-access-svfv2\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:19:59.336464 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:59.336431    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5a61eb0-8aa4-4004-a6ee-ed18256145e0" path="/var/lib/kubelet/pods/d5a61eb0-8aa4-4004-a6ee-ed18256145e0/volumes"
Sep 11 22:19:59.482846 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:59.482808    2587 generic.go:356] "Generic (PLEG): container finished" podID="d5a61eb0-8aa4-4004-a6ee-ed18256145e0" containerID="f9aae88ec610906324e88fdcb4ca8a823ee04faa8fb0a9287d199ce608b9d346" exitCode=137
Sep 11 22:19:59.482846 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:59.482848    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-0701c6"
Sep 11 22:19:59.483371 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:59.482912    2587 scope.go:117] "RemoveContainer" containerID="f9aae88ec610906324e88fdcb4ca8a823ee04faa8fb0a9287d199ce608b9d346"
Sep 11 22:19:59.491584 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:59.491563    2587 scope.go:117] "RemoveContainer" containerID="f9aae88ec610906324e88fdcb4ca8a823ee04faa8fb0a9287d199ce608b9d346"
Sep 11 22:19:59.491852 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:19:59.491835    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9aae88ec610906324e88fdcb4ca8a823ee04faa8fb0a9287d199ce608b9d346\": container with ID starting with f9aae88ec610906324e88fdcb4ca8a823ee04faa8fb0a9287d199ce608b9d346 not found: ID does not exist" containerID="f9aae88ec610906324e88fdcb4ca8a823ee04faa8fb0a9287d199ce608b9d346"
Sep 11 22:19:59.491905 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:19:59.491860    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9aae88ec610906324e88fdcb4ca8a823ee04faa8fb0a9287d199ce608b9d346"} err="failed to get container status \"f9aae88ec610906324e88fdcb4ca8a823ee04faa8fb0a9287d199ce608b9d346\": rpc error: code = NotFound desc = could not find container \"f9aae88ec610906324e88fdcb4ca8a823ee04faa8fb0a9287d199ce608b9d346\": container with ID starting with f9aae88ec610906324e88fdcb4ca8a823ee04faa8fb0a9287d199ce608b9d346 not found: ID does not exist"
Sep 11 22:20:00.661371 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:20:00.661342    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-dbd053"
Sep 11 22:20:00.730958 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:20:00.730874    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5npdd\" (UniqueName: \"kubernetes.io/projected/0d5e42bd-a325-4c11-95f0-857525e4697c-kube-api-access-5npdd\") pod \"0d5e42bd-a325-4c11-95f0-857525e4697c\" (UID: \"0d5e42bd-a325-4c11-95f0-857525e4697c\") "
Sep 11 22:20:00.732956 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:20:00.732925    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5e42bd-a325-4c11-95f0-857525e4697c-kube-api-access-5npdd" (OuterVolumeSpecName: "kube-api-access-5npdd") pod "0d5e42bd-a325-4c11-95f0-857525e4697c" (UID: "0d5e42bd-a325-4c11-95f0-857525e4697c"). InnerVolumeSpecName "kube-api-access-5npdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:20:00.831929 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:20:00.831893    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5npdd\" (UniqueName: \"kubernetes.io/projected/0d5e42bd-a325-4c11-95f0-857525e4697c-kube-api-access-5npdd\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:20:01.337475 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:20:01.337443    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d5e42bd-a325-4c11-95f0-857525e4697c" path="/var/lib/kubelet/pods/0d5e42bd-a325-4c11-95f0-857525e4697c/volumes"
Sep 11 22:20:01.469820 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:20:01.469785    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:20:01.495101 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:20:01.495062    2587 generic.go:356] "Generic (PLEG): container finished" podID="0d5e42bd-a325-4c11-95f0-857525e4697c" containerID="9a5631c5a15a046eedcf0290ebf2cea5d329e56b0cfaba59de408495b86c5b58" exitCode=137
Sep 11 22:20:01.495273 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:20:01.495111    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7684-dbd053"
Sep 11 22:20:01.495273 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:20:01.495153    2587 scope.go:117] "RemoveContainer" containerID="9a5631c5a15a046eedcf0290ebf2cea5d329e56b0cfaba59de408495b86c5b58"
Sep 11 22:20:01.503796 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:20:01.503777    2587 scope.go:117] "RemoveContainer" containerID="9a5631c5a15a046eedcf0290ebf2cea5d329e56b0cfaba59de408495b86c5b58"
Sep 11 22:20:01.504125 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:20:01.504102    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a5631c5a15a046eedcf0290ebf2cea5d329e56b0cfaba59de408495b86c5b58\": container with ID starting with 9a5631c5a15a046eedcf0290ebf2cea5d329e56b0cfaba59de408495b86c5b58 not found: ID does not exist" containerID="9a5631c5a15a046eedcf0290ebf2cea5d329e56b0cfaba59de408495b86c5b58"
Sep 11 22:20:01.504231 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:20:01.504137    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a5631c5a15a046eedcf0290ebf2cea5d329e56b0cfaba59de408495b86c5b58"} err="failed to get container status \"9a5631c5a15a046eedcf0290ebf2cea5d329e56b0cfaba59de408495b86c5b58\": rpc error: code = NotFound desc = could not find container \"9a5631c5a15a046eedcf0290ebf2cea5d329e56b0cfaba59de408495b86c5b58\": container with ID starting with 9a5631c5a15a046eedcf0290ebf2cea5d329e56b0cfaba59de408495b86c5b58 not found: ID does not exist"
Sep 11 22:20:12.638281 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:20:12.638245    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:20:45.669841 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:20:45.669803    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:21:03.367597 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:03.367500    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:21:13.166035 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:13.165999    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:21:21.700767 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.700732    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"]
Sep 11 22:21:21.701154 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.701130    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d5e42bd-a325-4c11-95f0-857525e4697c" containerName="test-curl-7684-dbd053"
Sep 11 22:21:21.701154 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.701141    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d5e42bd-a325-4c11-95f0-857525e4697c" containerName="test-curl-7684-dbd053"
Sep 11 22:21:21.701272 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.701209    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d5e42bd-a325-4c11-95f0-857525e4697c" containerName="test-curl-7684-dbd053"
Sep 11 22:21:21.704609 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.704581    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.708040 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.708019    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-6fe4c2-openshift-default-dockercfg-n57fj\""
Sep 11 22:21:21.715633 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.715200    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"]
Sep 11 22:21:21.718165 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.718137    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-istio-data\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.718295 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.718187    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42f08ea-111b-40c0-8abc-1138c00abe13-istio-podinfo\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.718295 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.718248    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-istio-envoy\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.718295 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.718284    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42f08ea-111b-40c0-8abc-1138c00abe13-istiod-ca-cert\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.718506 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.718312    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-credential-socket\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.718506 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.718361    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-workload-certs\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.718506 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.718392    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42f08ea-111b-40c0-8abc-1138c00abe13-istio-token\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.718506 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.718436    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-workload-socket\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.718506 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.718452    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vrvm\" (UniqueName: \"kubernetes.io/projected/b42f08ea-111b-40c0-8abc-1138c00abe13-kube-api-access-9vrvm\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.819919 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.819879    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-istio-data\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.820119 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.819941    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42f08ea-111b-40c0-8abc-1138c00abe13-istio-podinfo\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.820119 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.819974    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-istio-envoy\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.820119 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.819993    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42f08ea-111b-40c0-8abc-1138c00abe13-istiod-ca-cert\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.820119 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.820029    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-credential-socket\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.820119 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.820057    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-workload-certs\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.820119 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.820090    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42f08ea-111b-40c0-8abc-1138c00abe13-istio-token\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.820461 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.820150    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-workload-socket\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.820461 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.820173    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vrvm\" (UniqueName: \"kubernetes.io/projected/b42f08ea-111b-40c0-8abc-1138c00abe13-kube-api-access-9vrvm\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.820461 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.820363    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-istio-data\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.820622 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.820611    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-workload-certs\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.820817 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.820793    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-workload-socket\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.820887 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.820813    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42f08ea-111b-40c0-8abc-1138c00abe13-istiod-ca-cert\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.820949 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.820933    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-credential-socket\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.822556 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.822528    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-istio-envoy\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.823012 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.822990    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42f08ea-111b-40c0-8abc-1138c00abe13-istio-podinfo\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.828751 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.828722    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vrvm\" (UniqueName: \"kubernetes.io/projected/b42f08ea-111b-40c0-8abc-1138c00abe13-kube-api-access-9vrvm\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:21.829458 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:21.829439    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42f08ea-111b-40c0-8abc-1138c00abe13-istio-token\") pod \"e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") " pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:22.016784 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:22.016676    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:22.292505 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:22.292478    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"]
Sep 11 22:21:22.294250 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:21:22.294222    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42f08ea_111b_40c0_8abc_1138c00abe13.slice/crio-37dd033adc318e9e0f424b2b85416df45b23c44608d2e2bca79d7ee0fb9eb253 WatchSource:0}: Error finding container 37dd033adc318e9e0f424b2b85416df45b23c44608d2e2bca79d7ee0fb9eb253: Status 404 returned error can't find the container with id 37dd033adc318e9e0f424b2b85416df45b23c44608d2e2bca79d7ee0fb9eb253
Sep 11 22:21:22.296589 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:22.296557    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:21:22.296655 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:22.296625    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:21:22.296655 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:22.296651    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:21:22.914088 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:22.914048    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956" event={"ID":"b42f08ea-111b-40c0-8abc-1138c00abe13","Type":"ContainerStarted","Data":"bd885123503b8c6018c420b0b424d0e1ed4c07d7ddd6a648dd9ea343230d8f90"}
Sep 11 22:21:22.914531 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:22.914111    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956" event={"ID":"b42f08ea-111b-40c0-8abc-1138c00abe13","Type":"ContainerStarted","Data":"37dd033adc318e9e0f424b2b85416df45b23c44608d2e2bca79d7ee0fb9eb253"}
Sep 11 22:21:22.935679 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:22.935624    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956" podStartSLOduration=1.935607556 podStartE2EDuration="1.935607556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:21:22.934022821 +0000 UTC m=+1736.109099210" watchObservedRunningTime="2026-09-11 22:21:22.935607556 +0000 UTC m=+1736.110683926"
Sep 11 22:21:23.017518 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:23.017476    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:23.022890 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:23.022862    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:23.267897 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:23.267859    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:21:23.923988 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:23.923957    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:23.925024 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:23.925000    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:34.689186 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:34.689145    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:21:39.663681 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:39.663002    2587 patch_prober.go:28] interesting pod/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": read tcp 10.134.0.2:45224->10.134.0.78:15021: read: connection reset by peer" start-of-body=
Sep 11 22:21:39.663681 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:39.663078    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956" podUID="b42f08ea-111b-40c0-8abc-1138c00abe13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": read tcp 10.134.0.2:45224->10.134.0.78:15021: read: connection reset by peer"
Sep 11 22:21:40.011163 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:40.011126    2587 generic.go:356] "Generic (PLEG): container finished" podID="b42f08ea-111b-40c0-8abc-1138c00abe13" containerID="bd885123503b8c6018c420b0b424d0e1ed4c07d7ddd6a648dd9ea343230d8f90" exitCode=0
Sep 11 22:21:40.011391 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:40.011203    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956" event={"ID":"b42f08ea-111b-40c0-8abc-1138c00abe13","Type":"ContainerDied","Data":"bd885123503b8c6018c420b0b424d0e1ed4c07d7ddd6a648dd9ea343230d8f90"}
Sep 11 22:21:40.011655 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:40.011639    2587 scope.go:117] "RemoveContainer" containerID="bd885123503b8c6018c420b0b424d0e1ed4c07d7ddd6a648dd9ea343230d8f90"
Sep 11 22:21:40.012677 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:40.012651    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:21:40.012729 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:40.012720    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:21:40.012765 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:40.012749    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:21:41.018947 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:41.018912    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956" event={"ID":"b42f08ea-111b-40c0-8abc-1138c00abe13","Type":"ContainerStarted","Data":"1990a2b260005dc2a413bb39a669cb54ce73c71ba0a9b9741ca0f7c1f3212091"}
Sep 11 22:21:42.017618 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:42.017579    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:42.017820 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:42.017786    2587 patch_prober.go:28] interesting pod/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 11 22:21:42.017897 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:42.017842    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956" podUID="b42f08ea-111b-40c0-8abc-1138c00abe13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 11 22:21:43.017337 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:43.017301    2587 patch_prober.go:28] interesting pod/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 11 22:21:43.017733 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:43.017385    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956" podUID="b42f08ea-111b-40c0-8abc-1138c00abe13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 11 22:21:44.017222 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:44.017184    2587 patch_prober.go:28] interesting pod/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 11 22:21:44.017439 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:44.017243    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956" podUID="b42f08ea-111b-40c0-8abc-1138c00abe13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 11 22:21:45.017590 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:45.017555    2587 patch_prober.go:28] interesting pod/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 11 22:21:45.017994 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:45.017613    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956" podUID="b42f08ea-111b-40c0-8abc-1138c00abe13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 11 22:21:46.021033 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:46.020998    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:46.021460 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:46.021371    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:46.022076 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:46.022057    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:21:50.857229 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:50.857192    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"]
Sep 11 22:21:50.861500 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:50.861474    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:50.864159 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:50.864127    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-dbaf52-openshift-default-dockercfg-vbhhm\""
Sep 11 22:21:50.874925 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:50.874903    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"]
Sep 11 22:21:51.023323 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.023285    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-workload-certs\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.023323 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.023338    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istio-token\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.023626 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.023365    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istio-envoy\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.023626 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.023453    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-credential-socket\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.023626 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.023542    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istio-podinfo\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.023785 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.023709    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istio-data\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.023785 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.023758    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-workload-socket\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.023893 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.023783    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istiod-ca-cert\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.023893 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.023810    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6khxm\" (UniqueName: \"kubernetes.io/projected/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-kube-api-access-6khxm\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.124526 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.124429    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-credential-socket\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.124526 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.124498    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istio-podinfo\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.124769 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.124650    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istio-data\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.124769 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.124692    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-workload-socket\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.124769 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.124716    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istiod-ca-cert\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.124769 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.124741    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6khxm\" (UniqueName: \"kubernetes.io/projected/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-kube-api-access-6khxm\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.124987 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.124778    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-workload-certs\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.124987 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.124803    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istio-token\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.124987 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.124828    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istio-envoy\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.124987 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.124881    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-credential-socket\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.124987 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.124968    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istio-data\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.125250 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.125193    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-workload-socket\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.125315 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.125297    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-workload-certs\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.125965 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.125937    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istiod-ca-cert\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.127265 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.127240    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istio-envoy\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.127408 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.127374    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istio-podinfo\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.133037 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.133010    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istio-token\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.133512 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.133488    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6khxm\" (UniqueName: \"kubernetes.io/projected/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-kube-api-access-6khxm\") pod \"e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") " pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.175028 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.174996    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:51.466782 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.466654    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"]
Sep 11 22:21:51.469401 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:21:51.469375    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe1bdd03_f84d_4f8b_b9f3_dcf73c1ec0cd.slice/crio-f9ed2c3c5b9ee68f0a2c35764c58d961596019737eb53c3ae93ea1c97de0bc74 WatchSource:0}: Error finding container f9ed2c3c5b9ee68f0a2c35764c58d961596019737eb53c3ae93ea1c97de0bc74: Status 404 returned error can't find the container with id f9ed2c3c5b9ee68f0a2c35764c58d961596019737eb53c3ae93ea1c97de0bc74
Sep 11 22:21:51.471478 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.471445    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:21:51.471575 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.471523    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:21:51.471575 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:51.471551    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:21:52.094440 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:52.094404    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp" event={"ID":"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd","Type":"ContainerStarted","Data":"8f0011b193f7a23c065bc09d97884e4401c8ec9014c3ff5c7fb095b1e2d3aa46"}
Sep 11 22:21:52.094844 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:52.094454    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp" event={"ID":"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd","Type":"ContainerStarted","Data":"f9ed2c3c5b9ee68f0a2c35764c58d961596019737eb53c3ae93ea1c97de0bc74"}
Sep 11 22:21:52.114677 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:52.114628    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp" podStartSLOduration=2.114612479 podStartE2EDuration="2.114612479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:21:52.112975888 +0000 UTC m=+1765.288052278" watchObservedRunningTime="2026-09-11 22:21:52.114612479 +0000 UTC m=+1765.289688849"
Sep 11 22:21:52.175806 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:52.175772    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:52.180526 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:52.180500    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:53.099753 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:53.099728    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:21:53.100666 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:21:53.100646    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:22:01.476666 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:01.476627    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:22:12.205501 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:12.205467    2587 generic.go:356] "Generic (PLEG): container finished" podID="fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd" containerID="8f0011b193f7a23c065bc09d97884e4401c8ec9014c3ff5c7fb095b1e2d3aa46" exitCode=0
Sep 11 22:22:12.205944 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:12.205543    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp" event={"ID":"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd","Type":"ContainerDied","Data":"8f0011b193f7a23c065bc09d97884e4401c8ec9014c3ff5c7fb095b1e2d3aa46"}
Sep 11 22:22:12.205998 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:12.205950    2587 scope.go:117] "RemoveContainer" containerID="8f0011b193f7a23c065bc09d97884e4401c8ec9014c3ff5c7fb095b1e2d3aa46"
Sep 11 22:22:12.207130 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:12.207100    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:22:12.207200 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:12.207171    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:22:12.207238 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:12.207204    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:22:13.212634 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:13.212601    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp" event={"ID":"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd","Type":"ContainerStarted","Data":"c837f6c56659dc2a8892add9866b6944ea7c6e9e1c45e94a5559c874efc18e1a"}
Sep 11 22:22:14.175787 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:14.175741    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:22:14.176001 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:14.175901    2587 patch_prober.go:28] interesting pod/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 11 22:22:14.176001 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:14.175973    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp" podUID="fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 11 22:22:15.176144 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:15.176107    2587 patch_prober.go:28] interesting pod/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 11 22:22:15.176581 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:15.176169    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp" podUID="fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 11 22:22:16.175884 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:16.175846    2587 patch_prober.go:28] interesting pod/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 11 22:22:16.176064 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:16.175925    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp" podUID="fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 11 22:22:17.175708 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:17.175670    2587 patch_prober.go:28] interesting pod/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 11 22:22:17.176108 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:17.175740    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp" podUID="fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 11 22:22:18.176031 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:18.175984    2587 patch_prober.go:28] interesting pod/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Sep 11 22:22:18.176557 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:18.176058    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp" podUID="fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Sep 11 22:22:19.180016 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:19.179982    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:22:19.180432 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:19.180360    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:22:19.181263 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:19.181239    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:22:24.095481 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:24.095447    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-bsmwj_68c8f814-bf9c-4c1e-a642-741074aef1fd/payload-processing/0.log"
Sep 11 22:22:27.450265 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:27.450237    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw6qf_7ab8a820-c9ec-415c-bfaa-3c9744ac040a/ovn-acl-logging/0.log"
Sep 11 22:22:27.458972 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:27.458948    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw6qf_7ab8a820-c9ec-415c-bfaa-3c9744ac040a/ovn-acl-logging/0.log"
Sep 11 22:22:31.477266 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:31.477226    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:22:53.679385 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:22:53.679347    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:23:00.215108 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:00.215078    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-bsmwj_68c8f814-bf9c-4c1e-a642-741074aef1fd/payload-processing/0.log"
Sep 11 22:23:00.903751 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:00.903719    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"]
Sep 11 22:23:00.907639 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:00.907618    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:00.910497 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:00.910468    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-1688a8-openshift-default-dockercfg-f65nv\""
Sep 11 22:23:00.918415 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:00.918392    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"]
Sep 11 22:23:01.028969 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.028925    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-workload-socket\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.029182 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.028988    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istio-data\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.029182 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.029014    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istio-podinfo\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.029182 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.029072    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-workload-certs\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.029182 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.029099    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.029182 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.029147    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istio-token\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.029601 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.029196    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxbc7\" (UniqueName: \"kubernetes.io/projected/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-kube-api-access-mxbc7\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.029601 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.029260    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-credential-socket\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.029601 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.029371    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istio-envoy\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.130742 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.130705    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxbc7\" (UniqueName: \"kubernetes.io/projected/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-kube-api-access-mxbc7\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.130924 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.130764    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-credential-socket\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.130924 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.130783    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istio-envoy\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.130924 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.130824    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-workload-socket\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.130924 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.130853    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istio-data\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.130924 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.130876    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istio-podinfo\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.131184 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.130935    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-workload-certs\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.131184 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.130964    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.131184 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.131010    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istio-token\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.131385 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.131228    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-credential-socket\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.131385 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.131314    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-workload-socket\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.131498 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.131418    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istio-data\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.131498 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.131484    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-workload-certs\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.131825 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.131803    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.133259 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.133225    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istio-envoy\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.133391 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.133349    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istio-podinfo\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.138523 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.138497    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istio-token\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.138643 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.138615    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxbc7\" (UniqueName: \"kubernetes.io/projected/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-kube-api-access-mxbc7\") pod \"e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") " pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.219834 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.219786    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:01.503643 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.503613    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"]
Sep 11 22:23:01.505693 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:23:01.505667    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb8e7aad_8af6_4401_95ce_930cd1ebb8dd.slice/crio-3a88767139fce5218a2d90cf27f5cc13d86424f6e8cf64f784faa02c42a20b89 WatchSource:0}: Error finding container 3a88767139fce5218a2d90cf27f5cc13d86424f6e8cf64f784faa02c42a20b89: Status 404 returned error can't find the container with id 3a88767139fce5218a2d90cf27f5cc13d86424f6e8cf64f784faa02c42a20b89
Sep 11 22:23:01.508270 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.508240    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:23:01.508394 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.508315    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:23:01.508394 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:01.508356    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:23:02.480160 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:02.480123    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx" event={"ID":"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd","Type":"ContainerStarted","Data":"e55cfe81cec2d556f6242ba59a62d03aceab3f5fe4abf4c6d2db147e0e820dc0"}
Sep 11 22:23:02.480556 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:02.480189    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx" event={"ID":"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd","Type":"ContainerStarted","Data":"3a88767139fce5218a2d90cf27f5cc13d86424f6e8cf64f784faa02c42a20b89"}
Sep 11 22:23:02.502192 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:02.502144    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx" podStartSLOduration=2.5021272420000003 podStartE2EDuration="2.502127242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:23:02.500800602 +0000 UTC m=+1835.675876993" watchObservedRunningTime="2026-09-11 22:23:02.502127242 +0000 UTC m=+1835.677203611"
Sep 11 22:23:03.220079 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:03.220033    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:03.225560 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:03.225529    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:03.485300 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:03.485220    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:03.486661 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:03.486638    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:14.081463 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:14.081425    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:23:24.605591 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:24.605554    2587 generic.go:356] "Generic (PLEG): container finished" podID="fb8e7aad-8af6-4401-95ce-930cd1ebb8dd" containerID="e55cfe81cec2d556f6242ba59a62d03aceab3f5fe4abf4c6d2db147e0e820dc0" exitCode=0
Sep 11 22:23:24.606042 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:24.605605    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx" event={"ID":"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd","Type":"ContainerDied","Data":"e55cfe81cec2d556f6242ba59a62d03aceab3f5fe4abf4c6d2db147e0e820dc0"}
Sep 11 22:23:24.606042 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:24.605975    2587 scope.go:117] "RemoveContainer" containerID="e55cfe81cec2d556f6242ba59a62d03aceab3f5fe4abf4c6d2db147e0e820dc0"
Sep 11 22:23:24.607287 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:24.607260    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:23:24.607365 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:24.607355    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:23:24.607408 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:24.607386    2587 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 22:23:25.613165 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:25.613131    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx" event={"ID":"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd","Type":"ContainerStarted","Data":"a69bc230b917e27fa11389ee20a583aff09e0b943e4f714667bb59a10e455c45"}
Sep 11 22:23:26.220378 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:26.220297    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:26.220662 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:26.220633    2587 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 11 22:23:26.220746 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:26.220700    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx" podUID="fb8e7aad-8af6-4401-95ce-930cd1ebb8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 11 22:23:27.220864 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:27.220824    2587 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 11 22:23:27.222128 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:27.222095    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx" podUID="fb8e7aad-8af6-4401-95ce-930cd1ebb8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 11 22:23:28.220585 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:28.220541    2587 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 11 22:23:28.220902 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:28.220868    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx" podUID="fb8e7aad-8af6-4401-95ce-930cd1ebb8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 11 22:23:29.220958 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:29.220918    2587 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 11 22:23:29.221991 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:29.221955    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx" podUID="fb8e7aad-8af6-4401-95ce-930cd1ebb8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 11 22:23:30.220253 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:30.220210    2587 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 11 22:23:30.220662 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:30.220629    2587 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx" podUID="fb8e7aad-8af6-4401-95ce-930cd1ebb8dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 11 22:23:31.224109 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:31.224074    2587 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:31.224592 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:31.224373    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:31.225208 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:31.225191    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:50.869289 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:50.869205    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:23:52.624139 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:52.624105    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"]
Sep 11 22:23:52.624633 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:52.624387    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx" podUID="fb8e7aad-8af6-4401-95ce-930cd1ebb8dd" containerName="istio-proxy" containerID="cri-o://a69bc230b917e27fa11389ee20a583aff09e0b943e4f714667bb59a10e455c45" gracePeriod=30
Sep 11 22:23:57.794367 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:57.794215    2587 generic.go:356] "Generic (PLEG): container finished" podID="fb8e7aad-8af6-4401-95ce-930cd1ebb8dd" containerID="a69bc230b917e27fa11389ee20a583aff09e0b943e4f714667bb59a10e455c45" exitCode=0
Sep 11 22:23:57.794784 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:57.794379    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx" event={"ID":"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd","Type":"ContainerDied","Data":"a69bc230b917e27fa11389ee20a583aff09e0b943e4f714667bb59a10e455c45"}
Sep 11 22:23:57.794784 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:57.794441    2587 scope.go:117] "RemoveContainer" containerID="e55cfe81cec2d556f6242ba59a62d03aceab3f5fe4abf4c6d2db147e0e820dc0"
Sep 11 22:23:57.890396 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:57.890318    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:58.025555 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.025527    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-workload-certs\") pod \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") "
Sep 11 22:23:58.025555 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.025560    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-credential-socket\") pod \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") "
Sep 11 22:23:58.025811 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.025584    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istio-data\") pod \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") "
Sep 11 22:23:58.025811 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.025619    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istiod-ca-cert\") pod \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") "
Sep 11 22:23:58.025811 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.025640    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istio-token\") pod \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") "
Sep 11 22:23:58.025811 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.025660    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxbc7\" (UniqueName: \"kubernetes.io/projected/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-kube-api-access-mxbc7\") pod \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") "
Sep 11 22:23:58.025811 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.025688    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istio-envoy\") pod \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") "
Sep 11 22:23:58.025811 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.025743    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-workload-socket\") pod \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") "
Sep 11 22:23:58.025811 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.025803    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fb8e7aad-8af6-4401-95ce-930cd1ebb8dd" (UID: "fb8e7aad-8af6-4401-95ce-930cd1ebb8dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:23:58.026191 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.025826    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istio-podinfo\") pod \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\" (UID: \"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd\") "
Sep 11 22:23:58.026191 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.025879    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fb8e7aad-8af6-4401-95ce-930cd1ebb8dd" (UID: "fb8e7aad-8af6-4401-95ce-930cd1ebb8dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:23:58.026191 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.026004    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istio-data" (OuterVolumeSpecName: "istio-data") pod "fb8e7aad-8af6-4401-95ce-930cd1ebb8dd" (UID: "fb8e7aad-8af6-4401-95ce-930cd1ebb8dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:23:58.026191 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.026034    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fb8e7aad-8af6-4401-95ce-930cd1ebb8dd" (UID: "fb8e7aad-8af6-4401-95ce-930cd1ebb8dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:23:58.026407 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.026195    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fb8e7aad-8af6-4401-95ce-930cd1ebb8dd" (UID: "fb8e7aad-8af6-4401-95ce-930cd1ebb8dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:23:58.026407 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.026202    2587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-workload-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:23:58.026407 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.026253    2587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-workload-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:23:58.026407 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.026270    2587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-credential-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:23:58.026407 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.026284    2587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istio-data\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:23:58.028101 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.028076    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istio-token" (OuterVolumeSpecName: "istio-token") pod "fb8e7aad-8af6-4401-95ce-930cd1ebb8dd" (UID: "fb8e7aad-8af6-4401-95ce-930cd1ebb8dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:23:58.028214 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.028095    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-kube-api-access-mxbc7" (OuterVolumeSpecName: "kube-api-access-mxbc7") pod "fb8e7aad-8af6-4401-95ce-930cd1ebb8dd" (UID: "fb8e7aad-8af6-4401-95ce-930cd1ebb8dd"). InnerVolumeSpecName "kube-api-access-mxbc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:23:58.028405 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.028381    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fb8e7aad-8af6-4401-95ce-930cd1ebb8dd" (UID: "fb8e7aad-8af6-4401-95ce-930cd1ebb8dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:23:58.028464 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.028381    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fb8e7aad-8af6-4401-95ce-930cd1ebb8dd" (UID: "fb8e7aad-8af6-4401-95ce-930cd1ebb8dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:23:58.127075 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.127036    2587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istio-podinfo\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:23:58.127075 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.127073    2587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istiod-ca-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:23:58.127075 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.127087    2587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istio-token\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:23:58.127406 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.127101    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxbc7\" (UniqueName: \"kubernetes.io/projected/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-kube-api-access-mxbc7\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:23:58.127406 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.127115    2587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd-istio-envoy\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:23:58.801819 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.801783    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx" event={"ID":"fb8e7aad-8af6-4401-95ce-930cd1ebb8dd","Type":"ContainerDied","Data":"3a88767139fce5218a2d90cf27f5cc13d86424f6e8cf64f784faa02c42a20b89"}
Sep 11 22:23:58.802250 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.801844    2587 scope.go:117] "RemoveContainer" containerID="a69bc230b917e27fa11389ee20a583aff09e0b943e4f714667bb59a10e455c45"
Sep 11 22:23:58.802250 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.801798    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"
Sep 11 22:23:58.830033 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.829999    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"]
Sep 11 22:23:58.836277 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:58.836250    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1688a8-openshift-default-6b5bd94c76-tmsrx"]
Sep 11 22:23:59.336944 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:23:59.336907    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb8e7aad-8af6-4401-95ce-930cd1ebb8dd" path="/var/lib/kubelet/pods/fb8e7aad-8af6-4401-95ce-930cd1ebb8dd/volumes"
Sep 11 22:24:09.779808 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:24:09.779772    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:24:21.267847 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:24:21.267797    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:25:00.501625 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:00.501589    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"]
Sep 11 22:25:00.502173 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:00.501802    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956" podUID="b42f08ea-111b-40c0-8abc-1138c00abe13" containerName="istio-proxy" containerID="cri-o://1990a2b260005dc2a413bb39a669cb54ce73c71ba0a9b9741ca0f7c1f3212091" gracePeriod=30
Sep 11 22:25:01.023924 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:01.023887    2587 patch_prober.go:28] interesting pod/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 22:25:01.024076 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:01.023963    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956" podUID="b42f08ea-111b-40c0-8abc-1138c00abe13" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 22:25:05.767722 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:05.767651    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:25:05.913632 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:05.913599    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-credential-socket\") pod \"b42f08ea-111b-40c0-8abc-1138c00abe13\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") "
Sep 11 22:25:05.913805 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:05.913643    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42f08ea-111b-40c0-8abc-1138c00abe13-istiod-ca-cert\") pod \"b42f08ea-111b-40c0-8abc-1138c00abe13\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") "
Sep 11 22:25:05.913805 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:05.913686    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42f08ea-111b-40c0-8abc-1138c00abe13-istio-podinfo\") pod \"b42f08ea-111b-40c0-8abc-1138c00abe13\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") "
Sep 11 22:25:05.913805 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:05.913705    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42f08ea-111b-40c0-8abc-1138c00abe13-istio-token\") pod \"b42f08ea-111b-40c0-8abc-1138c00abe13\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") "
Sep 11 22:25:05.913958 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:05.913859    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-istio-data\") pod \"b42f08ea-111b-40c0-8abc-1138c00abe13\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") "
Sep 11 22:25:05.913958 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:05.913877    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b42f08ea-111b-40c0-8abc-1138c00abe13" (UID: "b42f08ea-111b-40c0-8abc-1138c00abe13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:25:05.913958 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:05.913903    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-istio-envoy\") pod \"b42f08ea-111b-40c0-8abc-1138c00abe13\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") "
Sep 11 22:25:05.913958 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:05.913932    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-workload-certs\") pod \"b42f08ea-111b-40c0-8abc-1138c00abe13\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") "
Sep 11 22:25:05.914169 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:05.913971    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-workload-socket\") pod \"b42f08ea-111b-40c0-8abc-1138c00abe13\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") "
Sep 11 22:25:05.914169 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:05.914017    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vrvm\" (UniqueName: \"kubernetes.io/projected/b42f08ea-111b-40c0-8abc-1138c00abe13-kube-api-access-9vrvm\") pod \"b42f08ea-111b-40c0-8abc-1138c00abe13\" (UID: \"b42f08ea-111b-40c0-8abc-1138c00abe13\") "
Sep 11 22:25:05.914169 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:05.914093    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-istio-data" (OuterVolumeSpecName: "istio-data") pod "b42f08ea-111b-40c0-8abc-1138c00abe13" (UID: "b42f08ea-111b-40c0-8abc-1138c00abe13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:25:05.914360 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:05.914177    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b42f08ea-111b-40c0-8abc-1138c00abe13" (UID: "b42f08ea-111b-40c0-8abc-1138c00abe13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:25:05.914360 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:05.914215    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b42f08ea-111b-40c0-8abc-1138c00abe13-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b42f08ea-111b-40c0-8abc-1138c00abe13" (UID: "b42f08ea-111b-40c0-8abc-1138c00abe13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:25:05.914464 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:05.914410    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b42f08ea-111b-40c0-8abc-1138c00abe13" (UID: "b42f08ea-111b-40c0-8abc-1138c00abe13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:25:05.914520 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:05.914480    2587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-credential-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:05.914520 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:05.914497    2587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42f08ea-111b-40c0-8abc-1138c00abe13-istiod-ca-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:05.914520 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:05.914511    2587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-istio-data\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:05.914644 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:05.914523    2587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-workload-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:05.914644 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:05.914531    2587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-workload-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:05.916555 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:05.916529    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42f08ea-111b-40c0-8abc-1138c00abe13-istio-token" (OuterVolumeSpecName: "istio-token") pod "b42f08ea-111b-40c0-8abc-1138c00abe13" (UID: "b42f08ea-111b-40c0-8abc-1138c00abe13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:25:05.916661 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:05.916550    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b42f08ea-111b-40c0-8abc-1138c00abe13-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b42f08ea-111b-40c0-8abc-1138c00abe13" (UID: "b42f08ea-111b-40c0-8abc-1138c00abe13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:25:05.916661 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:05.916562    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42f08ea-111b-40c0-8abc-1138c00abe13-kube-api-access-9vrvm" (OuterVolumeSpecName: "kube-api-access-9vrvm") pod "b42f08ea-111b-40c0-8abc-1138c00abe13" (UID: "b42f08ea-111b-40c0-8abc-1138c00abe13"). InnerVolumeSpecName "kube-api-access-9vrvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:25:05.916661 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:05.916551    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b42f08ea-111b-40c0-8abc-1138c00abe13" (UID: "b42f08ea-111b-40c0-8abc-1138c00abe13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:25:06.015233 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:06.015186    2587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42f08ea-111b-40c0-8abc-1138c00abe13-istio-envoy\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:06.015233 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:06.015228    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vrvm\" (UniqueName: \"kubernetes.io/projected/b42f08ea-111b-40c0-8abc-1138c00abe13-kube-api-access-9vrvm\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:06.015233 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:06.015239    2587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42f08ea-111b-40c0-8abc-1138c00abe13-istio-podinfo\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:06.015233 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:06.015249    2587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42f08ea-111b-40c0-8abc-1138c00abe13-istio-token\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:06.157771 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:06.157684    2587 generic.go:356] "Generic (PLEG): container finished" podID="b42f08ea-111b-40c0-8abc-1138c00abe13" containerID="1990a2b260005dc2a413bb39a669cb54ce73c71ba0a9b9741ca0f7c1f3212091" exitCode=0
Sep 11 22:25:06.157771 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:06.157727    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"
Sep 11 22:25:06.157949 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:06.157770    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956" event={"ID":"b42f08ea-111b-40c0-8abc-1138c00abe13","Type":"ContainerDied","Data":"1990a2b260005dc2a413bb39a669cb54ce73c71ba0a9b9741ca0f7c1f3212091"}
Sep 11 22:25:06.157949 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:06.157828    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956" event={"ID":"b42f08ea-111b-40c0-8abc-1138c00abe13","Type":"ContainerDied","Data":"37dd033adc318e9e0f424b2b85416df45b23c44608d2e2bca79d7ee0fb9eb253"}
Sep 11 22:25:06.157949 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:06.157851    2587 scope.go:117] "RemoveContainer" containerID="1990a2b260005dc2a413bb39a669cb54ce73c71ba0a9b9741ca0f7c1f3212091"
Sep 11 22:25:06.167305 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:06.167288    2587 scope.go:117] "RemoveContainer" containerID="bd885123503b8c6018c420b0b424d0e1ed4c07d7ddd6a648dd9ea343230d8f90"
Sep 11 22:25:06.175180 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:06.175163    2587 scope.go:117] "RemoveContainer" containerID="1990a2b260005dc2a413bb39a669cb54ce73c71ba0a9b9741ca0f7c1f3212091"
Sep 11 22:25:06.175499 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:25:06.175480    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1990a2b260005dc2a413bb39a669cb54ce73c71ba0a9b9741ca0f7c1f3212091\": container with ID starting with 1990a2b260005dc2a413bb39a669cb54ce73c71ba0a9b9741ca0f7c1f3212091 not found: ID does not exist" containerID="1990a2b260005dc2a413bb39a669cb54ce73c71ba0a9b9741ca0f7c1f3212091"
Sep 11 22:25:06.175564 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:06.175507    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1990a2b260005dc2a413bb39a669cb54ce73c71ba0a9b9741ca0f7c1f3212091"} err="failed to get container status \"1990a2b260005dc2a413bb39a669cb54ce73c71ba0a9b9741ca0f7c1f3212091\": rpc error: code = NotFound desc = could not find container \"1990a2b260005dc2a413bb39a669cb54ce73c71ba0a9b9741ca0f7c1f3212091\": container with ID starting with 1990a2b260005dc2a413bb39a669cb54ce73c71ba0a9b9741ca0f7c1f3212091 not found: ID does not exist"
Sep 11 22:25:06.175564 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:06.175531    2587 scope.go:117] "RemoveContainer" containerID="bd885123503b8c6018c420b0b424d0e1ed4c07d7ddd6a648dd9ea343230d8f90"
Sep 11 22:25:06.175798 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:25:06.175781    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd885123503b8c6018c420b0b424d0e1ed4c07d7ddd6a648dd9ea343230d8f90\": container with ID starting with bd885123503b8c6018c420b0b424d0e1ed4c07d7ddd6a648dd9ea343230d8f90 not found: ID does not exist" containerID="bd885123503b8c6018c420b0b424d0e1ed4c07d7ddd6a648dd9ea343230d8f90"
Sep 11 22:25:06.175850 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:06.175800    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd885123503b8c6018c420b0b424d0e1ed4c07d7ddd6a648dd9ea343230d8f90"} err="failed to get container status \"bd885123503b8c6018c420b0b424d0e1ed4c07d7ddd6a648dd9ea343230d8f90\": rpc error: code = NotFound desc = could not find container \"bd885123503b8c6018c420b0b424d0e1ed4c07d7ddd6a648dd9ea343230d8f90\": container with ID starting with bd885123503b8c6018c420b0b424d0e1ed4c07d7ddd6a648dd9ea343230d8f90 not found: ID does not exist"
Sep 11 22:25:06.179980 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:06.179957    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"]
Sep 11 22:25:06.183913 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:06.183891    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-6fe4c2-openshift-default-d69d4fb4b-6f956"]
Sep 11 22:25:07.336842 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:07.336809    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b42f08ea-111b-40c0-8abc-1138c00abe13" path="/var/lib/kubelet/pods/b42f08ea-111b-40c0-8abc-1138c00abe13/volumes"
Sep 11 22:25:14.469452 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:14.469418    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:25:18.249760 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:18.249724    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"]
Sep 11 22:25:18.250232 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:18.249943    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp" podUID="fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd" containerName="istio-proxy" containerID="cri-o://c837f6c56659dc2a8892add9866b6944ea7c6e9e1c45e94a5559c874efc18e1a" gracePeriod=30
Sep 11 22:25:19.183120 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:19.183086    2587 patch_prober.go:28] interesting pod/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 22:25:19.183288 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:19.183153    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp" podUID="fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 22:25:23.510057 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.509993    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:25:23.588563 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.588529    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istio-podinfo\") pod \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") "
Sep 11 22:25:23.588563 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.588576    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-credential-socket\") pod \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") "
Sep 11 22:25:23.588838 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.588606    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istio-data\") pod \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") "
Sep 11 22:25:23.588838 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.588650    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6khxm\" (UniqueName: \"kubernetes.io/projected/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-kube-api-access-6khxm\") pod \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") "
Sep 11 22:25:23.588838 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.588721    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istiod-ca-cert\") pod \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") "
Sep 11 22:25:23.588838 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.588750    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-workload-socket\") pod \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") "
Sep 11 22:25:23.588838 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.588777    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istio-envoy\") pod \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") "
Sep 11 22:25:23.588838 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.588839    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-workload-certs\") pod \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") "
Sep 11 22:25:23.589159 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.588876    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istio-token\") pod \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\" (UID: \"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd\") "
Sep 11 22:25:23.589159 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.588873    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd" (UID: "fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:25:23.589159 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.589138    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd" (UID: "fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:25:23.589355 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.589155    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd" (UID: "fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:25:23.589355 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.589195    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd" (UID: "fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:25:23.589355 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.589202    2587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-credential-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:23.589519 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.589361    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istio-data" (OuterVolumeSpecName: "istio-data") pod "fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd" (UID: "fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:25:23.590910 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.590876    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd" (UID: "fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:25:23.591035 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.590931    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-kube-api-access-6khxm" (OuterVolumeSpecName: "kube-api-access-6khxm") pod "fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd" (UID: "fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd"). InnerVolumeSpecName "kube-api-access-6khxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:25:23.591035 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.590934    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd" (UID: "fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:25:23.591114 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.591065    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istio-token" (OuterVolumeSpecName: "istio-token") pod "fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd" (UID: "fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:25:23.690357 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.690299    2587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-workload-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:23.690357 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.690360    2587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istio-token\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:23.690565 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.690375    2587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istio-podinfo\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:23.690565 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.690386    2587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istio-data\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:23.690565 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.690397    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6khxm\" (UniqueName: \"kubernetes.io/projected/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-kube-api-access-6khxm\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:23.690565 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.690407    2587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istiod-ca-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:23.690565 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.690415    2587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-workload-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:23.690565 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:23.690423    2587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd-istio-envoy\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:24.260979 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:24.260944    2587 generic.go:356] "Generic (PLEG): container finished" podID="fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd" containerID="c837f6c56659dc2a8892add9866b6944ea7c6e9e1c45e94a5559c874efc18e1a" exitCode=0
Sep 11 22:25:24.261149 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:24.260992    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"
Sep 11 22:25:24.261149 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:24.260996    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp" event={"ID":"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd","Type":"ContainerDied","Data":"c837f6c56659dc2a8892add9866b6944ea7c6e9e1c45e94a5559c874efc18e1a"}
Sep 11 22:25:24.261149 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:24.261113    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp" event={"ID":"fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd","Type":"ContainerDied","Data":"f9ed2c3c5b9ee68f0a2c35764c58d961596019737eb53c3ae93ea1c97de0bc74"}
Sep 11 22:25:24.261149 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:24.261139    2587 scope.go:117] "RemoveContainer" containerID="c837f6c56659dc2a8892add9866b6944ea7c6e9e1c45e94a5559c874efc18e1a"
Sep 11 22:25:24.271346 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:24.271059    2587 scope.go:117] "RemoveContainer" containerID="8f0011b193f7a23c065bc09d97884e4401c8ec9014c3ff5c7fb095b1e2d3aa46"
Sep 11 22:25:24.279489 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:24.279466    2587 scope.go:117] "RemoveContainer" containerID="c837f6c56659dc2a8892add9866b6944ea7c6e9e1c45e94a5559c874efc18e1a"
Sep 11 22:25:24.279759 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:25:24.279734    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c837f6c56659dc2a8892add9866b6944ea7c6e9e1c45e94a5559c874efc18e1a\": container with ID starting with c837f6c56659dc2a8892add9866b6944ea7c6e9e1c45e94a5559c874efc18e1a not found: ID does not exist" containerID="c837f6c56659dc2a8892add9866b6944ea7c6e9e1c45e94a5559c874efc18e1a"
Sep 11 22:25:24.279812 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:24.279773    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c837f6c56659dc2a8892add9866b6944ea7c6e9e1c45e94a5559c874efc18e1a"} err="failed to get container status \"c837f6c56659dc2a8892add9866b6944ea7c6e9e1c45e94a5559c874efc18e1a\": rpc error: code = NotFound desc = could not find container \"c837f6c56659dc2a8892add9866b6944ea7c6e9e1c45e94a5559c874efc18e1a\": container with ID starting with c837f6c56659dc2a8892add9866b6944ea7c6e9e1c45e94a5559c874efc18e1a not found: ID does not exist"
Sep 11 22:25:24.279812 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:24.279795    2587 scope.go:117] "RemoveContainer" containerID="8f0011b193f7a23c065bc09d97884e4401c8ec9014c3ff5c7fb095b1e2d3aa46"
Sep 11 22:25:24.280083 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:25:24.280066    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f0011b193f7a23c065bc09d97884e4401c8ec9014c3ff5c7fb095b1e2d3aa46\": container with ID starting with 8f0011b193f7a23c065bc09d97884e4401c8ec9014c3ff5c7fb095b1e2d3aa46 not found: ID does not exist" containerID="8f0011b193f7a23c065bc09d97884e4401c8ec9014c3ff5c7fb095b1e2d3aa46"
Sep 11 22:25:24.280131 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:24.280087    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0011b193f7a23c065bc09d97884e4401c8ec9014c3ff5c7fb095b1e2d3aa46"} err="failed to get container status \"8f0011b193f7a23c065bc09d97884e4401c8ec9014c3ff5c7fb095b1e2d3aa46\": rpc error: code = NotFound desc = could not find container \"8f0011b193f7a23c065bc09d97884e4401c8ec9014c3ff5c7fb095b1e2d3aa46\": container with ID starting with 8f0011b193f7a23c065bc09d97884e4401c8ec9014c3ff5c7fb095b1e2d3aa46 not found: ID does not exist"
Sep 11 22:25:24.285178 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:24.285154    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"]
Sep 11 22:25:24.290630 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:24.290604    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-dbaf52-openshift-default-6f79688f7b-8rhkp"]
Sep 11 22:25:25.336443 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:25.336410    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd" path="/var/lib/kubelet/pods/fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd/volumes"
Sep 11 22:25:29.124499 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:29.124458    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5"]
Sep 11 22:25:29.125058 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:29.124720    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5" podUID="06f1362b-5ccc-4ca7-8239-e5982339df97" containerName="maas-api" containerID="cri-o://530219b2842440a7d4e8ab03b775a06bf7e35f857c37d1112a471f8352264b06" gracePeriod=30
Sep 11 22:25:29.288851 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:29.288818    2587 generic.go:356] "Generic (PLEG): container finished" podID="06f1362b-5ccc-4ca7-8239-e5982339df97" containerID="530219b2842440a7d4e8ab03b775a06bf7e35f857c37d1112a471f8352264b06" exitCode=0
Sep 11 22:25:29.289013 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:29.288895    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5" event={"ID":"06f1362b-5ccc-4ca7-8239-e5982339df97","Type":"ContainerDied","Data":"530219b2842440a7d4e8ab03b775a06bf7e35f857c37d1112a471f8352264b06"}
Sep 11 22:25:29.375857 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:29.375788    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5"
Sep 11 22:25:29.446527 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:29.446487    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnk6k\" (UniqueName: \"kubernetes.io/projected/06f1362b-5ccc-4ca7-8239-e5982339df97-kube-api-access-nnk6k\") pod \"06f1362b-5ccc-4ca7-8239-e5982339df97\" (UID: \"06f1362b-5ccc-4ca7-8239-e5982339df97\") "
Sep 11 22:25:29.446728 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:29.446616    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06f1362b-5ccc-4ca7-8239-e5982339df97-metrics-certs\") pod \"06f1362b-5ccc-4ca7-8239-e5982339df97\" (UID: \"06f1362b-5ccc-4ca7-8239-e5982339df97\") "
Sep 11 22:25:29.446728 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:29.446665    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06f1362b-5ccc-4ca7-8239-e5982339df97-maas-api-tls\") pod \"06f1362b-5ccc-4ca7-8239-e5982339df97\" (UID: \"06f1362b-5ccc-4ca7-8239-e5982339df97\") "
Sep 11 22:25:29.446728 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:29.446696    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06f1362b-5ccc-4ca7-8239-e5982339df97-openshift-service-ca\") pod \"06f1362b-5ccc-4ca7-8239-e5982339df97\" (UID: \"06f1362b-5ccc-4ca7-8239-e5982339df97\") "
Sep 11 22:25:29.447142 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:29.447112    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06f1362b-5ccc-4ca7-8239-e5982339df97-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "06f1362b-5ccc-4ca7-8239-e5982339df97" (UID: "06f1362b-5ccc-4ca7-8239-e5982339df97"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:25:29.449281 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:29.449246    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06f1362b-5ccc-4ca7-8239-e5982339df97-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "06f1362b-5ccc-4ca7-8239-e5982339df97" (UID: "06f1362b-5ccc-4ca7-8239-e5982339df97"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:25:29.449281 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:29.449267    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06f1362b-5ccc-4ca7-8239-e5982339df97-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "06f1362b-5ccc-4ca7-8239-e5982339df97" (UID: "06f1362b-5ccc-4ca7-8239-e5982339df97"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:25:29.449456 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:29.449302    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f1362b-5ccc-4ca7-8239-e5982339df97-kube-api-access-nnk6k" (OuterVolumeSpecName: "kube-api-access-nnk6k") pod "06f1362b-5ccc-4ca7-8239-e5982339df97" (UID: "06f1362b-5ccc-4ca7-8239-e5982339df97"). InnerVolumeSpecName "kube-api-access-nnk6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:25:29.548022 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:29.547982    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnk6k\" (UniqueName: \"kubernetes.io/projected/06f1362b-5ccc-4ca7-8239-e5982339df97-kube-api-access-nnk6k\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:29.548022 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:29.548024    2587 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06f1362b-5ccc-4ca7-8239-e5982339df97-metrics-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:29.548233 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:29.548038    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06f1362b-5ccc-4ca7-8239-e5982339df97-maas-api-tls\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:29.548233 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:29.548051    2587 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06f1362b-5ccc-4ca7-8239-e5982339df97-openshift-service-ca\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:30.295309 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:30.295276    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5" event={"ID":"06f1362b-5ccc-4ca7-8239-e5982339df97","Type":"ContainerDied","Data":"edfed889476adff5a846d074807661ec6c0c6e7b1c751162ad2468f9404cabda"}
Sep 11 22:25:30.295309 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:30.295308    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5"
Sep 11 22:25:30.295883 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:30.295352    2587 scope.go:117] "RemoveContainer" containerID="530219b2842440a7d4e8ab03b775a06bf7e35f857c37d1112a471f8352264b06"
Sep 11 22:25:30.320045 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:30.320017    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5"]
Sep 11 22:25:30.324425 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:30.324401    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d09c96-767f86db76-j8qw5"]
Sep 11 22:25:31.335942 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:31.335910    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06f1362b-5ccc-4ca7-8239-e5982339df97" path="/var/lib/kubelet/pods/06f1362b-5ccc-4ca7-8239-e5982339df97/volumes"
Sep 11 22:25:32.074365 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:32.074319    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:25:36.616190 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:36.616154    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"]
Sep 11 22:25:36.616717 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:36.616443    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j" podUID="64f7d428-457a-4b96-bb21-266305d3cab7" containerName="istio-proxy" containerID="cri-o://c033d8f50ff1c0ff5636388ffa9e7be508fd1cdd52bfe6571b5ee1d670da3946" gracePeriod=30
Sep 11 22:25:38.181443 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:38.181401    2587 patch_prober.go:28] interesting pod/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 22:25:38.181835 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:38.181462    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j" podUID="64f7d428-457a-4b96-bb21-266305d3cab7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 22:25:41.869794 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:41.869765    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:25:41.962607 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:41.962571    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64f7d428-457a-4b96-bb21-266305d3cab7-istio-podinfo\") pod \"64f7d428-457a-4b96-bb21-266305d3cab7\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") "
Sep 11 22:25:41.962797 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:41.962626    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-workload-socket\") pod \"64f7d428-457a-4b96-bb21-266305d3cab7\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") "
Sep 11 22:25:41.962797 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:41.962659    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-istio-envoy\") pod \"64f7d428-457a-4b96-bb21-266305d3cab7\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") "
Sep 11 22:25:41.962797 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:41.962696    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-workload-certs\") pod \"64f7d428-457a-4b96-bb21-266305d3cab7\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") "
Sep 11 22:25:41.962797 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:41.962724    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64f7d428-457a-4b96-bb21-266305d3cab7-istio-token\") pod \"64f7d428-457a-4b96-bb21-266305d3cab7\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") "
Sep 11 22:25:41.962797 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:41.962750    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64f7d428-457a-4b96-bb21-266305d3cab7-istiod-ca-cert\") pod \"64f7d428-457a-4b96-bb21-266305d3cab7\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") "
Sep 11 22:25:41.962797 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:41.962783    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpb29\" (UniqueName: \"kubernetes.io/projected/64f7d428-457a-4b96-bb21-266305d3cab7-kube-api-access-mpb29\") pod \"64f7d428-457a-4b96-bb21-266305d3cab7\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") "
Sep 11 22:25:41.963080 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:41.962812    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-credential-socket\") pod \"64f7d428-457a-4b96-bb21-266305d3cab7\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") "
Sep 11 22:25:41.963080 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:41.962927    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-istio-data\") pod \"64f7d428-457a-4b96-bb21-266305d3cab7\" (UID: \"64f7d428-457a-4b96-bb21-266305d3cab7\") "
Sep 11 22:25:41.963080 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:41.962994    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "64f7d428-457a-4b96-bb21-266305d3cab7" (UID: "64f7d428-457a-4b96-bb21-266305d3cab7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:25:41.963080 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:41.963014    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "64f7d428-457a-4b96-bb21-266305d3cab7" (UID: "64f7d428-457a-4b96-bb21-266305d3cab7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:25:41.963278 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:41.963194    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "64f7d428-457a-4b96-bb21-266305d3cab7" (UID: "64f7d428-457a-4b96-bb21-266305d3cab7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:25:41.963278 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:41.963208    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-istio-data" (OuterVolumeSpecName: "istio-data") pod "64f7d428-457a-4b96-bb21-266305d3cab7" (UID: "64f7d428-457a-4b96-bb21-266305d3cab7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:25:41.963415 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:41.963315    2587 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-istio-data\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:41.963415 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:41.963344    2587 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-workload-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:41.963415 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:41.963353    2587 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-workload-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:41.963415 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:41.963364    2587 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-credential-socket\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:41.963640 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:41.963575    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f7d428-457a-4b96-bb21-266305d3cab7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "64f7d428-457a-4b96-bb21-266305d3cab7" (UID: "64f7d428-457a-4b96-bb21-266305d3cab7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:25:41.964918 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:41.964894    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64f7d428-457a-4b96-bb21-266305d3cab7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "64f7d428-457a-4b96-bb21-266305d3cab7" (UID: "64f7d428-457a-4b96-bb21-266305d3cab7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 22:25:41.964918 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:41.964901    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "64f7d428-457a-4b96-bb21-266305d3cab7" (UID: "64f7d428-457a-4b96-bb21-266305d3cab7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 22:25:41.965061 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:41.964992    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f7d428-457a-4b96-bb21-266305d3cab7-istio-token" (OuterVolumeSpecName: "istio-token") pod "64f7d428-457a-4b96-bb21-266305d3cab7" (UID: "64f7d428-457a-4b96-bb21-266305d3cab7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:25:41.965061 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:41.965020    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f7d428-457a-4b96-bb21-266305d3cab7-kube-api-access-mpb29" (OuterVolumeSpecName: "kube-api-access-mpb29") pod "64f7d428-457a-4b96-bb21-266305d3cab7" (UID: "64f7d428-457a-4b96-bb21-266305d3cab7"). InnerVolumeSpecName "kube-api-access-mpb29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:25:42.064266 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:42.064231    2587 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64f7d428-457a-4b96-bb21-266305d3cab7-istio-podinfo\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:42.064266 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:42.064260    2587 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64f7d428-457a-4b96-bb21-266305d3cab7-istio-envoy\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:42.064266 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:42.064269    2587 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64f7d428-457a-4b96-bb21-266305d3cab7-istio-token\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:42.064522 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:42.064277    2587 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64f7d428-457a-4b96-bb21-266305d3cab7-istiod-ca-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:42.064522 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:42.064286    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpb29\" (UniqueName: \"kubernetes.io/projected/64f7d428-457a-4b96-bb21-266305d3cab7-kube-api-access-mpb29\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:42.358803 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:42.358703    2587 generic.go:356] "Generic (PLEG): container finished" podID="64f7d428-457a-4b96-bb21-266305d3cab7" containerID="c033d8f50ff1c0ff5636388ffa9e7be508fd1cdd52bfe6571b5ee1d670da3946" exitCode=0
Sep 11 22:25:42.358803 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:42.358755    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j" event={"ID":"64f7d428-457a-4b96-bb21-266305d3cab7","Type":"ContainerDied","Data":"c033d8f50ff1c0ff5636388ffa9e7be508fd1cdd52bfe6571b5ee1d670da3946"}
Sep 11 22:25:42.358803 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:42.358761    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"
Sep 11 22:25:42.358803 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:42.358798    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j" event={"ID":"64f7d428-457a-4b96-bb21-266305d3cab7","Type":"ContainerDied","Data":"7c6c3609a018637c702ca52d07f82fbd058199fa2f5be96a9587598c0c446f6d"}
Sep 11 22:25:42.359132 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:42.358819    2587 scope.go:117] "RemoveContainer" containerID="c033d8f50ff1c0ff5636388ffa9e7be508fd1cdd52bfe6571b5ee1d670da3946"
Sep 11 22:25:42.368027 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:42.368009    2587 scope.go:117] "RemoveContainer" containerID="a604650a1a4e11f913267cc4f25d3acee64b5313d8912e393d2d9ba8110c0409"
Sep 11 22:25:42.376304 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:42.376284    2587 scope.go:117] "RemoveContainer" containerID="c033d8f50ff1c0ff5636388ffa9e7be508fd1cdd52bfe6571b5ee1d670da3946"
Sep 11 22:25:42.376614 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:25:42.376594    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c033d8f50ff1c0ff5636388ffa9e7be508fd1cdd52bfe6571b5ee1d670da3946\": container with ID starting with c033d8f50ff1c0ff5636388ffa9e7be508fd1cdd52bfe6571b5ee1d670da3946 not found: ID does not exist" containerID="c033d8f50ff1c0ff5636388ffa9e7be508fd1cdd52bfe6571b5ee1d670da3946"
Sep 11 22:25:42.376662 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:42.376623    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c033d8f50ff1c0ff5636388ffa9e7be508fd1cdd52bfe6571b5ee1d670da3946"} err="failed to get container status \"c033d8f50ff1c0ff5636388ffa9e7be508fd1cdd52bfe6571b5ee1d670da3946\": rpc error: code = NotFound desc = could not find container \"c033d8f50ff1c0ff5636388ffa9e7be508fd1cdd52bfe6571b5ee1d670da3946\": container with ID starting with c033d8f50ff1c0ff5636388ffa9e7be508fd1cdd52bfe6571b5ee1d670da3946 not found: ID does not exist"
Sep 11 22:25:42.376662 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:42.376639    2587 scope.go:117] "RemoveContainer" containerID="a604650a1a4e11f913267cc4f25d3acee64b5313d8912e393d2d9ba8110c0409"
Sep 11 22:25:42.376888 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:25:42.376869    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a604650a1a4e11f913267cc4f25d3acee64b5313d8912e393d2d9ba8110c0409\": container with ID starting with a604650a1a4e11f913267cc4f25d3acee64b5313d8912e393d2d9ba8110c0409 not found: ID does not exist" containerID="a604650a1a4e11f913267cc4f25d3acee64b5313d8912e393d2d9ba8110c0409"
Sep 11 22:25:42.376951 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:42.376898    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a604650a1a4e11f913267cc4f25d3acee64b5313d8912e393d2d9ba8110c0409"} err="failed to get container status \"a604650a1a4e11f913267cc4f25d3acee64b5313d8912e393d2d9ba8110c0409\": rpc error: code = NotFound desc = could not find container \"a604650a1a4e11f913267cc4f25d3acee64b5313d8912e393d2d9ba8110c0409\": container with ID starting with a604650a1a4e11f913267cc4f25d3acee64b5313d8912e393d2d9ba8110c0409 not found: ID does not exist"
Sep 11 22:25:42.380513 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:42.380488    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"]
Sep 11 22:25:42.384463 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:42.384441    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d09c96-openshift-default-7f5bf5ffc8-b4j5j"]
Sep 11 22:25:43.335670 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:43.335638    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64f7d428-457a-4b96-bb21-266305d3cab7" path="/var/lib/kubelet/pods/64f7d428-457a-4b96-bb21-266305d3cab7/volumes"
Sep 11 22:25:47.394393 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:47.394357    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp"]
Sep 11 22:25:47.394881 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:47.394583    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp" podUID="b82f3465-9969-4110-88a4-dfec9d1bc7a0" containerName="maas-api" containerID="cri-o://65ca001cf5026a29612214878bcb0a15a2d56aa89049aca0d6bd7d73aa5c2323" gracePeriod=30
Sep 11 22:25:47.649888 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:47.649826    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp"
Sep 11 22:25:47.823636 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:47.823608    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b82f3465-9969-4110-88a4-dfec9d1bc7a0-metrics-certs\") pod \"b82f3465-9969-4110-88a4-dfec9d1bc7a0\" (UID: \"b82f3465-9969-4110-88a4-dfec9d1bc7a0\") "
Sep 11 22:25:47.823636 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:47.823641    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b82f3465-9969-4110-88a4-dfec9d1bc7a0-maas-api-tls\") pod \"b82f3465-9969-4110-88a4-dfec9d1bc7a0\" (UID: \"b82f3465-9969-4110-88a4-dfec9d1bc7a0\") "
Sep 11 22:25:47.823904 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:47.823694    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgpst\" (UniqueName: \"kubernetes.io/projected/b82f3465-9969-4110-88a4-dfec9d1bc7a0-kube-api-access-kgpst\") pod \"b82f3465-9969-4110-88a4-dfec9d1bc7a0\" (UID: \"b82f3465-9969-4110-88a4-dfec9d1bc7a0\") "
Sep 11 22:25:47.823904 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:47.823810    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b82f3465-9969-4110-88a4-dfec9d1bc7a0-openshift-service-ca\") pod \"b82f3465-9969-4110-88a4-dfec9d1bc7a0\" (UID: \"b82f3465-9969-4110-88a4-dfec9d1bc7a0\") "
Sep 11 22:25:47.824477 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:47.824440    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b82f3465-9969-4110-88a4-dfec9d1bc7a0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b82f3465-9969-4110-88a4-dfec9d1bc7a0" (UID: "b82f3465-9969-4110-88a4-dfec9d1bc7a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:25:47.826005 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:47.825971    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b82f3465-9969-4110-88a4-dfec9d1bc7a0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b82f3465-9969-4110-88a4-dfec9d1bc7a0" (UID: "b82f3465-9969-4110-88a4-dfec9d1bc7a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:25:47.826252 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:47.826229    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b82f3465-9969-4110-88a4-dfec9d1bc7a0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b82f3465-9969-4110-88a4-dfec9d1bc7a0" (UID: "b82f3465-9969-4110-88a4-dfec9d1bc7a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:25:47.826389 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:47.826297    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b82f3465-9969-4110-88a4-dfec9d1bc7a0-kube-api-access-kgpst" (OuterVolumeSpecName: "kube-api-access-kgpst") pod "b82f3465-9969-4110-88a4-dfec9d1bc7a0" (UID: "b82f3465-9969-4110-88a4-dfec9d1bc7a0"). InnerVolumeSpecName "kube-api-access-kgpst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:25:47.925144 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:47.925110    2587 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b82f3465-9969-4110-88a4-dfec9d1bc7a0-openshift-service-ca\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:47.925144 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:47.925139    2587 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b82f3465-9969-4110-88a4-dfec9d1bc7a0-metrics-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:47.925346 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:47.925151    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b82f3465-9969-4110-88a4-dfec9d1bc7a0-maas-api-tls\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:47.925346 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:47.925159    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgpst\" (UniqueName: \"kubernetes.io/projected/b82f3465-9969-4110-88a4-dfec9d1bc7a0-kube-api-access-kgpst\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:48.392050 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:48.391953    2587 generic.go:356] "Generic (PLEG): container finished" podID="b82f3465-9969-4110-88a4-dfec9d1bc7a0" containerID="65ca001cf5026a29612214878bcb0a15a2d56aa89049aca0d6bd7d73aa5c2323" exitCode=0
Sep 11 22:25:48.392050 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:48.392008    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp"
Sep 11 22:25:48.392050 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:48.392008    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp" event={"ID":"b82f3465-9969-4110-88a4-dfec9d1bc7a0","Type":"ContainerDied","Data":"65ca001cf5026a29612214878bcb0a15a2d56aa89049aca0d6bd7d73aa5c2323"}
Sep 11 22:25:48.392357 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:48.392058    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp" event={"ID":"b82f3465-9969-4110-88a4-dfec9d1bc7a0","Type":"ContainerDied","Data":"802949714e1465193ad44b82592a7e6da9efbae4c8648fd51b647c1da43c8ed7"}
Sep 11 22:25:48.392357 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:48.392078    2587 scope.go:117] "RemoveContainer" containerID="65ca001cf5026a29612214878bcb0a15a2d56aa89049aca0d6bd7d73aa5c2323"
Sep 11 22:25:48.401555 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:48.401365    2587 scope.go:117] "RemoveContainer" containerID="65ca001cf5026a29612214878bcb0a15a2d56aa89049aca0d6bd7d73aa5c2323"
Sep 11 22:25:48.401769 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:25:48.401656    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65ca001cf5026a29612214878bcb0a15a2d56aa89049aca0d6bd7d73aa5c2323\": container with ID starting with 65ca001cf5026a29612214878bcb0a15a2d56aa89049aca0d6bd7d73aa5c2323 not found: ID does not exist" containerID="65ca001cf5026a29612214878bcb0a15a2d56aa89049aca0d6bd7d73aa5c2323"
Sep 11 22:25:48.401769 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:48.401682    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65ca001cf5026a29612214878bcb0a15a2d56aa89049aca0d6bd7d73aa5c2323"} err="failed to get container status \"65ca001cf5026a29612214878bcb0a15a2d56aa89049aca0d6bd7d73aa5c2323\": rpc error: code = NotFound desc = could not find container \"65ca001cf5026a29612214878bcb0a15a2d56aa89049aca0d6bd7d73aa5c2323\": container with ID starting with 65ca001cf5026a29612214878bcb0a15a2d56aa89049aca0d6bd7d73aa5c2323 not found: ID does not exist"
Sep 11 22:25:48.413514 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:48.413481    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp"]
Sep 11 22:25:48.417260 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:48.417229    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-966527-794966fc65-lg8kp"]
Sep 11 22:25:49.335745 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:49.335704    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b82f3465-9969-4110-88a4-dfec9d1bc7a0" path="/var/lib/kubelet/pods/b82f3465-9969-4110-88a4-dfec9d1bc7a0/volumes"
Sep 11 22:25:50.064802 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:50.064770    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:25:56.906225 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:56.906186    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fcd74f577-d2b6p"]
Sep 11 22:25:56.906740 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:56.906470    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5fcd74f577-d2b6p" podUID="828b0b6b-f551-4582-8920-0ea68e295a50" containerName="manager" containerID="cri-o://32a865fe74dc645d77e06e62f870d89aaa070fe6b312505dcc35926bfbbb2564" gracePeriod=30
Sep 11 22:25:57.147134 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:57.147108    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fcd74f577-d2b6p"
Sep 11 22:25:57.208462 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:57.208431    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52kjz\" (UniqueName: \"kubernetes.io/projected/828b0b6b-f551-4582-8920-0ea68e295a50-kube-api-access-52kjz\") pod \"828b0b6b-f551-4582-8920-0ea68e295a50\" (UID: \"828b0b6b-f551-4582-8920-0ea68e295a50\") "
Sep 11 22:25:57.208649 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:57.208475    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/828b0b6b-f551-4582-8920-0ea68e295a50-metrics-certs\") pod \"828b0b6b-f551-4582-8920-0ea68e295a50\" (UID: \"828b0b6b-f551-4582-8920-0ea68e295a50\") "
Sep 11 22:25:57.208649 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:57.208568    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/828b0b6b-f551-4582-8920-0ea68e295a50-webhook-cert\") pod \"828b0b6b-f551-4582-8920-0ea68e295a50\" (UID: \"828b0b6b-f551-4582-8920-0ea68e295a50\") "
Sep 11 22:25:57.210741 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:57.210690    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/828b0b6b-f551-4582-8920-0ea68e295a50-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "828b0b6b-f551-4582-8920-0ea68e295a50" (UID: "828b0b6b-f551-4582-8920-0ea68e295a50"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:25:57.211082 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:57.211061    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828b0b6b-f551-4582-8920-0ea68e295a50-kube-api-access-52kjz" (OuterVolumeSpecName: "kube-api-access-52kjz") pod "828b0b6b-f551-4582-8920-0ea68e295a50" (UID: "828b0b6b-f551-4582-8920-0ea68e295a50"). InnerVolumeSpecName "kube-api-access-52kjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:25:57.211180 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:57.211059    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/828b0b6b-f551-4582-8920-0ea68e295a50-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "828b0b6b-f551-4582-8920-0ea68e295a50" (UID: "828b0b6b-f551-4582-8920-0ea68e295a50"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:25:57.309636 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:57.309602    2587 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/828b0b6b-f551-4582-8920-0ea68e295a50-webhook-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:57.309636 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:57.309632    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52kjz\" (UniqueName: \"kubernetes.io/projected/828b0b6b-f551-4582-8920-0ea68e295a50-kube-api-access-52kjz\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:57.309636 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:57.309643    2587 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/828b0b6b-f551-4582-8920-0ea68e295a50-metrics-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:25:57.437617 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:57.437581    2587 generic.go:356] "Generic (PLEG): container finished" podID="828b0b6b-f551-4582-8920-0ea68e295a50" containerID="32a865fe74dc645d77e06e62f870d89aaa070fe6b312505dcc35926bfbbb2564" exitCode=0
Sep 11 22:25:57.437799 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:57.437626    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fcd74f577-d2b6p"
Sep 11 22:25:57.437799 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:57.437633    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fcd74f577-d2b6p" event={"ID":"828b0b6b-f551-4582-8920-0ea68e295a50","Type":"ContainerDied","Data":"32a865fe74dc645d77e06e62f870d89aaa070fe6b312505dcc35926bfbbb2564"}
Sep 11 22:25:57.437799 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:57.437673    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fcd74f577-d2b6p" event={"ID":"828b0b6b-f551-4582-8920-0ea68e295a50","Type":"ContainerDied","Data":"ee5b176e355e9e1b8ef4638cc57fe6d209347464e4ee2c732e4a94e17272b1b7"}
Sep 11 22:25:57.437799 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:57.437693    2587 scope.go:117] "RemoveContainer" containerID="32a865fe74dc645d77e06e62f870d89aaa070fe6b312505dcc35926bfbbb2564"
Sep 11 22:25:57.453143 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:57.453115    2587 scope.go:117] "RemoveContainer" containerID="32a865fe74dc645d77e06e62f870d89aaa070fe6b312505dcc35926bfbbb2564"
Sep 11 22:25:57.453488 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:25:57.453463    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32a865fe74dc645d77e06e62f870d89aaa070fe6b312505dcc35926bfbbb2564\": container with ID starting with 32a865fe74dc645d77e06e62f870d89aaa070fe6b312505dcc35926bfbbb2564 not found: ID does not exist" containerID="32a865fe74dc645d77e06e62f870d89aaa070fe6b312505dcc35926bfbbb2564"
Sep 11 22:25:57.453592 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:57.453494    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32a865fe74dc645d77e06e62f870d89aaa070fe6b312505dcc35926bfbbb2564"} err="failed to get container status \"32a865fe74dc645d77e06e62f870d89aaa070fe6b312505dcc35926bfbbb2564\": rpc error: code = NotFound desc = could not find container \"32a865fe74dc645d77e06e62f870d89aaa070fe6b312505dcc35926bfbbb2564\": container with ID starting with 32a865fe74dc645d77e06e62f870d89aaa070fe6b312505dcc35926bfbbb2564 not found: ID does not exist"
Sep 11 22:25:57.458316 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:57.458285    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fcd74f577-d2b6p"]
Sep 11 22:25:57.462346 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:57.462275    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5fcd74f577-d2b6p"]
Sep 11 22:25:58.970436 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:58.970394    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:25:59.338279 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:25:59.338201    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="828b0b6b-f551-4582-8920-0ea68e295a50" path="/var/lib/kubelet/pods/828b0b6b-f551-4582-8920-0ea68e295a50/volumes"
Sep 11 22:26:58.243269 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243178    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fcd74f577-nqnr7"]
Sep 11 22:26:58.243778 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243578    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb8e7aad-8af6-4401-95ce-930cd1ebb8dd" containerName="istio-proxy"
Sep 11 22:26:58.243778 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243591    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb8e7aad-8af6-4401-95ce-930cd1ebb8dd" containerName="istio-proxy"
Sep 11 22:26:58.243778 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243599    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="828b0b6b-f551-4582-8920-0ea68e295a50" containerName="manager"
Sep 11 22:26:58.243778 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243604    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="828b0b6b-f551-4582-8920-0ea68e295a50" containerName="manager"
Sep 11 22:26:58.243778 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243614    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06f1362b-5ccc-4ca7-8239-e5982339df97" containerName="maas-api"
Sep 11 22:26:58.243778 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243620    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="06f1362b-5ccc-4ca7-8239-e5982339df97" containerName="maas-api"
Sep 11 22:26:58.243778 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243627    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64f7d428-457a-4b96-bb21-266305d3cab7" containerName="istio-proxy"
Sep 11 22:26:58.243778 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243632    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="64f7d428-457a-4b96-bb21-266305d3cab7" containerName="istio-proxy"
Sep 11 22:26:58.243778 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243639    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b42f08ea-111b-40c0-8abc-1138c00abe13" containerName="istio-proxy"
Sep 11 22:26:58.243778 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243644    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="b42f08ea-111b-40c0-8abc-1138c00abe13" containerName="istio-proxy"
Sep 11 22:26:58.243778 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243652    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b82f3465-9969-4110-88a4-dfec9d1bc7a0" containerName="maas-api"
Sep 11 22:26:58.243778 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243657    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="b82f3465-9969-4110-88a4-dfec9d1bc7a0" containerName="maas-api"
Sep 11 22:26:58.243778 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243665    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64f7d428-457a-4b96-bb21-266305d3cab7" containerName="istio-proxy"
Sep 11 22:26:58.243778 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243671    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="64f7d428-457a-4b96-bb21-266305d3cab7" containerName="istio-proxy"
Sep 11 22:26:58.243778 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243680    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b42f08ea-111b-40c0-8abc-1138c00abe13" containerName="istio-proxy"
Sep 11 22:26:58.243778 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243685    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="b42f08ea-111b-40c0-8abc-1138c00abe13" containerName="istio-proxy"
Sep 11 22:26:58.243778 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243698    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd" containerName="istio-proxy"
Sep 11 22:26:58.243778 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243702    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd" containerName="istio-proxy"
Sep 11 22:26:58.243778 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243710    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd" containerName="istio-proxy"
Sep 11 22:26:58.243778 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243715    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd" containerName="istio-proxy"
Sep 11 22:26:58.243778 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243785    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="64f7d428-457a-4b96-bb21-266305d3cab7" containerName="istio-proxy"
Sep 11 22:26:58.244438 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243794    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="b42f08ea-111b-40c0-8abc-1138c00abe13" containerName="istio-proxy"
Sep 11 22:26:58.244438 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243799    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd" containerName="istio-proxy"
Sep 11 22:26:58.244438 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243806    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe1bdd03-f84d-4f8b-b9f3-dcf73c1ec0cd" containerName="istio-proxy"
Sep 11 22:26:58.244438 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243811    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb8e7aad-8af6-4401-95ce-930cd1ebb8dd" containerName="istio-proxy"
Sep 11 22:26:58.244438 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243817    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="828b0b6b-f551-4582-8920-0ea68e295a50" containerName="manager"
Sep 11 22:26:58.244438 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243821    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb8e7aad-8af6-4401-95ce-930cd1ebb8dd" containerName="istio-proxy"
Sep 11 22:26:58.244438 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243827    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="06f1362b-5ccc-4ca7-8239-e5982339df97" containerName="maas-api"
Sep 11 22:26:58.244438 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.243833    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="b82f3465-9969-4110-88a4-dfec9d1bc7a0" containerName="maas-api"
Sep 11 22:26:58.246985 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.246966    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fcd74f577-nqnr7"
Sep 11 22:26:58.251479 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.251441    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kwx56\""
Sep 11 22:26:58.251661 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.251474    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 22:26:58.251731 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.251683    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 22:26:58.251731 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.251691    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 22:26:58.256501 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.256469    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fcd74f577-nqnr7"]
Sep 11 22:26:58.352245 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.352200    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xq76\" (UniqueName: \"kubernetes.io/projected/b34a5e26-0def-4c0b-8785-9472ff4a2dd8-kube-api-access-7xq76\") pod \"maas-controller-5fcd74f577-nqnr7\" (UID: \"b34a5e26-0def-4c0b-8785-9472ff4a2dd8\") " pod="opendatahub/maas-controller-5fcd74f577-nqnr7"
Sep 11 22:26:58.352245 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.352248    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b34a5e26-0def-4c0b-8785-9472ff4a2dd8-metrics-certs\") pod \"maas-controller-5fcd74f577-nqnr7\" (UID: \"b34a5e26-0def-4c0b-8785-9472ff4a2dd8\") " pod="opendatahub/maas-controller-5fcd74f577-nqnr7"
Sep 11 22:26:58.352523 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.352306    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b34a5e26-0def-4c0b-8785-9472ff4a2dd8-webhook-cert\") pod \"maas-controller-5fcd74f577-nqnr7\" (UID: \"b34a5e26-0def-4c0b-8785-9472ff4a2dd8\") " pod="opendatahub/maas-controller-5fcd74f577-nqnr7"
Sep 11 22:26:58.453201 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.453160    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xq76\" (UniqueName: \"kubernetes.io/projected/b34a5e26-0def-4c0b-8785-9472ff4a2dd8-kube-api-access-7xq76\") pod \"maas-controller-5fcd74f577-nqnr7\" (UID: \"b34a5e26-0def-4c0b-8785-9472ff4a2dd8\") " pod="opendatahub/maas-controller-5fcd74f577-nqnr7"
Sep 11 22:26:58.453201 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.453203    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b34a5e26-0def-4c0b-8785-9472ff4a2dd8-metrics-certs\") pod \"maas-controller-5fcd74f577-nqnr7\" (UID: \"b34a5e26-0def-4c0b-8785-9472ff4a2dd8\") " pod="opendatahub/maas-controller-5fcd74f577-nqnr7"
Sep 11 22:26:58.453488 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.453222    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b34a5e26-0def-4c0b-8785-9472ff4a2dd8-webhook-cert\") pod \"maas-controller-5fcd74f577-nqnr7\" (UID: \"b34a5e26-0def-4c0b-8785-9472ff4a2dd8\") " pod="opendatahub/maas-controller-5fcd74f577-nqnr7"
Sep 11 22:26:58.455788 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.455763    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b34a5e26-0def-4c0b-8785-9472ff4a2dd8-metrics-certs\") pod \"maas-controller-5fcd74f577-nqnr7\" (UID: \"b34a5e26-0def-4c0b-8785-9472ff4a2dd8\") " pod="opendatahub/maas-controller-5fcd74f577-nqnr7"
Sep 11 22:26:58.455914 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.455880    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b34a5e26-0def-4c0b-8785-9472ff4a2dd8-webhook-cert\") pod \"maas-controller-5fcd74f577-nqnr7\" (UID: \"b34a5e26-0def-4c0b-8785-9472ff4a2dd8\") " pod="opendatahub/maas-controller-5fcd74f577-nqnr7"
Sep 11 22:26:58.462066 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.462036    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xq76\" (UniqueName: \"kubernetes.io/projected/b34a5e26-0def-4c0b-8785-9472ff4a2dd8-kube-api-access-7xq76\") pod \"maas-controller-5fcd74f577-nqnr7\" (UID: \"b34a5e26-0def-4c0b-8785-9472ff4a2dd8\") " pod="opendatahub/maas-controller-5fcd74f577-nqnr7"
Sep 11 22:26:58.559633 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.559537    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fcd74f577-nqnr7"
Sep 11 22:26:58.833073 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.833036    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fcd74f577-nqnr7"]
Sep 11 22:26:58.834245 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:26:58.834221    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb34a5e26_0def_4c0b_8785_9472ff4a2dd8.slice/crio-764d13732fe9bea2b8718bc24ff8c18b0f8f965e1871065cdc0591054c9fb5ad WatchSource:0}: Error finding container 764d13732fe9bea2b8718bc24ff8c18b0f8f965e1871065cdc0591054c9fb5ad: Status 404 returned error can't find the container with id 764d13732fe9bea2b8718bc24ff8c18b0f8f965e1871065cdc0591054c9fb5ad
Sep 11 22:26:58.835887 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:58.835867    2587 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 22:26:59.749676 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:59.749631    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fcd74f577-nqnr7" event={"ID":"b34a5e26-0def-4c0b-8785-9472ff4a2dd8","Type":"ContainerStarted","Data":"011eb7518907344b3eb7b71db3f2265fc891eb7d4a6872487608b4bed5bd29e6"}
Sep 11 22:26:59.750128 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:59.749710    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fcd74f577-nqnr7" event={"ID":"b34a5e26-0def-4c0b-8785-9472ff4a2dd8","Type":"ContainerStarted","Data":"764d13732fe9bea2b8718bc24ff8c18b0f8f965e1871065cdc0591054c9fb5ad"}
Sep 11 22:26:59.750128 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:59.749751    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fcd74f577-nqnr7"
Sep 11 22:26:59.794636 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:26:59.794560    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5fcd74f577-nqnr7" podStartSLOduration=1.237194999 podStartE2EDuration="1.794543139s" totalImagesPullingTime="557.34814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:26:59.79306745 +0000 UTC m=+2072.968143811" watchObservedRunningTime="2026-09-11 22:26:59.794543139 +0000 UTC m=+2072.969619509"
Sep 11 22:27:10.757740 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:10.757707    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fcd74f577-nqnr7"
Sep 11 22:27:19.669780 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:19.669745    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:27:20.152909 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:20.152827    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fcd74f577-nqnr7"]
Sep 11 22:27:20.153058 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:20.153037    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5fcd74f577-nqnr7" podUID="b34a5e26-0def-4c0b-8785-9472ff4a2dd8" containerName="manager" containerID="cri-o://011eb7518907344b3eb7b71db3f2265fc891eb7d4a6872487608b4bed5bd29e6" gracePeriod=30
Sep 11 22:27:20.400646 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:20.400619    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fcd74f577-nqnr7"
Sep 11 22:27:20.568660 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:20.568601    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b34a5e26-0def-4c0b-8785-9472ff4a2dd8-webhook-cert\") pod \"b34a5e26-0def-4c0b-8785-9472ff4a2dd8\" (UID: \"b34a5e26-0def-4c0b-8785-9472ff4a2dd8\") "
Sep 11 22:27:20.568856 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:20.568814    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b34a5e26-0def-4c0b-8785-9472ff4a2dd8-metrics-certs\") pod \"b34a5e26-0def-4c0b-8785-9472ff4a2dd8\" (UID: \"b34a5e26-0def-4c0b-8785-9472ff4a2dd8\") "
Sep 11 22:27:20.568950 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:20.568904    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xq76\" (UniqueName: \"kubernetes.io/projected/b34a5e26-0def-4c0b-8785-9472ff4a2dd8-kube-api-access-7xq76\") pod \"b34a5e26-0def-4c0b-8785-9472ff4a2dd8\" (UID: \"b34a5e26-0def-4c0b-8785-9472ff4a2dd8\") "
Sep 11 22:27:20.570825 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:20.570783    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b34a5e26-0def-4c0b-8785-9472ff4a2dd8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b34a5e26-0def-4c0b-8785-9472ff4a2dd8" (UID: "b34a5e26-0def-4c0b-8785-9472ff4a2dd8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:27:20.571425 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:20.571395    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b34a5e26-0def-4c0b-8785-9472ff4a2dd8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b34a5e26-0def-4c0b-8785-9472ff4a2dd8" (UID: "b34a5e26-0def-4c0b-8785-9472ff4a2dd8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:27:20.571561 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:20.571475    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34a5e26-0def-4c0b-8785-9472ff4a2dd8-kube-api-access-7xq76" (OuterVolumeSpecName: "kube-api-access-7xq76") pod "b34a5e26-0def-4c0b-8785-9472ff4a2dd8" (UID: "b34a5e26-0def-4c0b-8785-9472ff4a2dd8"). InnerVolumeSpecName "kube-api-access-7xq76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:27:20.670064 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:20.670008    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xq76\" (UniqueName: \"kubernetes.io/projected/b34a5e26-0def-4c0b-8785-9472ff4a2dd8-kube-api-access-7xq76\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:27:20.670064 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:20.670068    2587 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b34a5e26-0def-4c0b-8785-9472ff4a2dd8-webhook-cert\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:27:20.670497 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:20.670085    2587 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b34a5e26-0def-4c0b-8785-9472ff4a2dd8-metrics-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:27:20.853708 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:20.853612    2587 generic.go:356] "Generic (PLEG): container finished" podID="b34a5e26-0def-4c0b-8785-9472ff4a2dd8" containerID="011eb7518907344b3eb7b71db3f2265fc891eb7d4a6872487608b4bed5bd29e6" exitCode=0
Sep 11 22:27:20.853708 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:20.853671    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fcd74f577-nqnr7"
Sep 11 22:27:20.853924 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:20.853682    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fcd74f577-nqnr7" event={"ID":"b34a5e26-0def-4c0b-8785-9472ff4a2dd8","Type":"ContainerDied","Data":"011eb7518907344b3eb7b71db3f2265fc891eb7d4a6872487608b4bed5bd29e6"}
Sep 11 22:27:20.853924 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:20.853816    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fcd74f577-nqnr7" event={"ID":"b34a5e26-0def-4c0b-8785-9472ff4a2dd8","Type":"ContainerDied","Data":"764d13732fe9bea2b8718bc24ff8c18b0f8f965e1871065cdc0591054c9fb5ad"}
Sep 11 22:27:20.853924 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:20.853836    2587 scope.go:117] "RemoveContainer" containerID="011eb7518907344b3eb7b71db3f2265fc891eb7d4a6872487608b4bed5bd29e6"
Sep 11 22:27:20.863727 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:20.863708    2587 scope.go:117] "RemoveContainer" containerID="011eb7518907344b3eb7b71db3f2265fc891eb7d4a6872487608b4bed5bd29e6"
Sep 11 22:27:20.863990 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:27:20.863970    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"011eb7518907344b3eb7b71db3f2265fc891eb7d4a6872487608b4bed5bd29e6\": container with ID starting with 011eb7518907344b3eb7b71db3f2265fc891eb7d4a6872487608b4bed5bd29e6 not found: ID does not exist" containerID="011eb7518907344b3eb7b71db3f2265fc891eb7d4a6872487608b4bed5bd29e6"
Sep 11 22:27:20.864064 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:20.863998    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"011eb7518907344b3eb7b71db3f2265fc891eb7d4a6872487608b4bed5bd29e6"} err="failed to get container status \"011eb7518907344b3eb7b71db3f2265fc891eb7d4a6872487608b4bed5bd29e6\": rpc error: code = NotFound desc = could not find container \"011eb7518907344b3eb7b71db3f2265fc891eb7d4a6872487608b4bed5bd29e6\": container with ID starting with 011eb7518907344b3eb7b71db3f2265fc891eb7d4a6872487608b4bed5bd29e6 not found: ID does not exist"
Sep 11 22:27:20.875968 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:20.875944    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fcd74f577-nqnr7"]
Sep 11 22:27:20.877668 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:20.877649    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5fcd74f577-nqnr7"]
Sep 11 22:27:21.335881 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:21.335853    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b34a5e26-0def-4c0b-8785-9472ff4a2dd8" path="/var/lib/kubelet/pods/b34a5e26-0def-4c0b-8785-9472ff4a2dd8/volumes"
Sep 11 22:27:21.669655 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:21.669623    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fcd74f577-5hnlp"]
Sep 11 22:27:21.670000 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:21.669987    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb8e7aad-8af6-4401-95ce-930cd1ebb8dd" containerName="istio-proxy"
Sep 11 22:27:21.670049 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:21.670001    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb8e7aad-8af6-4401-95ce-930cd1ebb8dd" containerName="istio-proxy"
Sep 11 22:27:21.670049 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:21.670013    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b34a5e26-0def-4c0b-8785-9472ff4a2dd8" containerName="manager"
Sep 11 22:27:21.670049 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:21.670019    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="b34a5e26-0def-4c0b-8785-9472ff4a2dd8" containerName="manager"
Sep 11 22:27:21.670393 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:21.670075    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="64f7d428-457a-4b96-bb21-266305d3cab7" containerName="istio-proxy"
Sep 11 22:27:21.670393 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:21.670087    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="b42f08ea-111b-40c0-8abc-1138c00abe13" containerName="istio-proxy"
Sep 11 22:27:21.670393 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:21.670095    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="b34a5e26-0def-4c0b-8785-9472ff4a2dd8" containerName="manager"
Sep 11 22:27:21.674501 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:21.674481    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fcd74f577-5hnlp"
Sep 11 22:27:21.678591 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:21.678539    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 22:27:21.678591 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:21.678553    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 22:27:21.678591 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:21.678576    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 22:27:21.678845 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:21.678639    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-kwx56\""
Sep 11 22:27:21.681603 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:21.681518    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fcd74f577-5hnlp"]
Sep 11 22:27:21.780265 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:21.780219    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/515308b9-9cbd-4828-aee1-48f55f701bac-metrics-certs\") pod \"maas-controller-5fcd74f577-5hnlp\" (UID: \"515308b9-9cbd-4828-aee1-48f55f701bac\") " pod="opendatahub/maas-controller-5fcd74f577-5hnlp"
Sep 11 22:27:21.780265 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:21.780266    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5mzx\" (UniqueName: \"kubernetes.io/projected/515308b9-9cbd-4828-aee1-48f55f701bac-kube-api-access-s5mzx\") pod \"maas-controller-5fcd74f577-5hnlp\" (UID: \"515308b9-9cbd-4828-aee1-48f55f701bac\") " pod="opendatahub/maas-controller-5fcd74f577-5hnlp"
Sep 11 22:27:21.780507 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:21.780370    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/515308b9-9cbd-4828-aee1-48f55f701bac-webhook-cert\") pod \"maas-controller-5fcd74f577-5hnlp\" (UID: \"515308b9-9cbd-4828-aee1-48f55f701bac\") " pod="opendatahub/maas-controller-5fcd74f577-5hnlp"
Sep 11 22:27:21.881426 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:21.881389    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/515308b9-9cbd-4828-aee1-48f55f701bac-metrics-certs\") pod \"maas-controller-5fcd74f577-5hnlp\" (UID: \"515308b9-9cbd-4828-aee1-48f55f701bac\") " pod="opendatahub/maas-controller-5fcd74f577-5hnlp"
Sep 11 22:27:21.881426 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:21.881428    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5mzx\" (UniqueName: \"kubernetes.io/projected/515308b9-9cbd-4828-aee1-48f55f701bac-kube-api-access-s5mzx\") pod \"maas-controller-5fcd74f577-5hnlp\" (UID: \"515308b9-9cbd-4828-aee1-48f55f701bac\") " pod="opendatahub/maas-controller-5fcd74f577-5hnlp"
Sep 11 22:27:21.881694 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:21.881494    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/515308b9-9cbd-4828-aee1-48f55f701bac-webhook-cert\") pod \"maas-controller-5fcd74f577-5hnlp\" (UID: \"515308b9-9cbd-4828-aee1-48f55f701bac\") " pod="opendatahub/maas-controller-5fcd74f577-5hnlp"
Sep 11 22:27:21.884032 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:21.884009    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/515308b9-9cbd-4828-aee1-48f55f701bac-metrics-certs\") pod \"maas-controller-5fcd74f577-5hnlp\" (UID: \"515308b9-9cbd-4828-aee1-48f55f701bac\") " pod="opendatahub/maas-controller-5fcd74f577-5hnlp"
Sep 11 22:27:21.884138 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:21.884009    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/515308b9-9cbd-4828-aee1-48f55f701bac-webhook-cert\") pod \"maas-controller-5fcd74f577-5hnlp\" (UID: \"515308b9-9cbd-4828-aee1-48f55f701bac\") " pod="opendatahub/maas-controller-5fcd74f577-5hnlp"
Sep 11 22:27:21.890200 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:21.890173    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5mzx\" (UniqueName: \"kubernetes.io/projected/515308b9-9cbd-4828-aee1-48f55f701bac-kube-api-access-s5mzx\") pod \"maas-controller-5fcd74f577-5hnlp\" (UID: \"515308b9-9cbd-4828-aee1-48f55f701bac\") " pod="opendatahub/maas-controller-5fcd74f577-5hnlp"
Sep 11 22:27:21.986352 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:21.986241    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fcd74f577-5hnlp"
Sep 11 22:27:22.257519 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:22.257494    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fcd74f577-5hnlp"]
Sep 11 22:27:22.259425 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:27:22.259397    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod515308b9_9cbd_4828_aee1_48f55f701bac.slice/crio-a0645d3da81cc182fc4611bffcdee30478136cfc984dc5e9f278a564dcf94d81 WatchSource:0}: Error finding container a0645d3da81cc182fc4611bffcdee30478136cfc984dc5e9f278a564dcf94d81: Status 404 returned error can't find the container with id a0645d3da81cc182fc4611bffcdee30478136cfc984dc5e9f278a564dcf94d81
Sep 11 22:27:22.866728 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:22.866692    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fcd74f577-5hnlp" event={"ID":"515308b9-9cbd-4828-aee1-48f55f701bac","Type":"ContainerStarted","Data":"9d382221cb873a8e1f6cc4b31b9a7be74e6e65157f845b248bc02ffe03870cfa"}
Sep 11 22:27:22.867202 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:22.866754    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fcd74f577-5hnlp"
Sep 11 22:27:22.867202 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:22.866778    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fcd74f577-5hnlp" event={"ID":"515308b9-9cbd-4828-aee1-48f55f701bac","Type":"ContainerStarted","Data":"a0645d3da81cc182fc4611bffcdee30478136cfc984dc5e9f278a564dcf94d81"}
Sep 11 22:27:22.884714 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:22.884665    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5fcd74f577-5hnlp" podStartSLOduration=1.359124537 podStartE2EDuration="1.884648011s" totalImagesPullingTime="525.523474ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:27:22.883010031 +0000 UTC m=+2096.058086421" watchObservedRunningTime="2026-09-11 22:27:22.884648011 +0000 UTC m=+2096.059724380"
Sep 11 22:27:27.479279 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:27.479247    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw6qf_7ab8a820-c9ec-415c-bfaa-3c9744ac040a/ovn-acl-logging/0.log"
Sep 11 22:27:27.489024 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:27.489002    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lw6qf_7ab8a820-c9ec-415c-bfaa-3c9744ac040a/ovn-acl-logging/0.log"
Sep 11 22:27:33.874043 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:33.874002    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fcd74f577-5hnlp"
Sep 11 22:27:45.674802 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:45.674764    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:27:53.867980 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:27:53.867937    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:28:02.175154 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:28:02.175115    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:28:15.774376 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:28:15.774339    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:28:21.473960 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:28:21.473928    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:29:47.365111 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:29:47.365074    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:29:54.880375 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:29:54.880273    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:30:03.875484 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:30:03.875447    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:30:13.134046 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:30:13.134006    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:30:51.880002 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:30:51.879961    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:30:59.474085 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:30:59.474042    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:31:06.583698 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:06.583660    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:31:07.030036 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:07.029995    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6945bc8b6b-tcl57"]
Sep 11 22:31:07.034376 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:07.034320    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-tcl57"
Sep 11 22:31:07.046248 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:07.046216    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6945bc8b6b-tcl57"]
Sep 11 22:31:07.112101 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:07.112064    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68a08d2e-2a17-4002-93d1-7b2c08aadbb7-maas-api-tls\") pod \"maas-api-6945bc8b6b-tcl57\" (UID: \"68a08d2e-2a17-4002-93d1-7b2c08aadbb7\") " pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-tcl57"
Sep 11 22:31:07.112281 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:07.112118    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68a08d2e-2a17-4002-93d1-7b2c08aadbb7-metrics-certs\") pod \"maas-api-6945bc8b6b-tcl57\" (UID: \"68a08d2e-2a17-4002-93d1-7b2c08aadbb7\") " pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-tcl57"
Sep 11 22:31:07.112281 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:07.112193    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68a08d2e-2a17-4002-93d1-7b2c08aadbb7-openshift-service-ca\") pod \"maas-api-6945bc8b6b-tcl57\" (UID: \"68a08d2e-2a17-4002-93d1-7b2c08aadbb7\") " pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-tcl57"
Sep 11 22:31:07.112281 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:07.112255    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb8xh\" (UniqueName: \"kubernetes.io/projected/68a08d2e-2a17-4002-93d1-7b2c08aadbb7-kube-api-access-tb8xh\") pod \"maas-api-6945bc8b6b-tcl57\" (UID: \"68a08d2e-2a17-4002-93d1-7b2c08aadbb7\") " pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-tcl57"
Sep 11 22:31:07.213195 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:07.213163    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68a08d2e-2a17-4002-93d1-7b2c08aadbb7-maas-api-tls\") pod \"maas-api-6945bc8b6b-tcl57\" (UID: \"68a08d2e-2a17-4002-93d1-7b2c08aadbb7\") " pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-tcl57"
Sep 11 22:31:07.213423 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:07.213221    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68a08d2e-2a17-4002-93d1-7b2c08aadbb7-metrics-certs\") pod \"maas-api-6945bc8b6b-tcl57\" (UID: \"68a08d2e-2a17-4002-93d1-7b2c08aadbb7\") " pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-tcl57"
Sep 11 22:31:07.213423 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:07.213257    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68a08d2e-2a17-4002-93d1-7b2c08aadbb7-openshift-service-ca\") pod \"maas-api-6945bc8b6b-tcl57\" (UID: \"68a08d2e-2a17-4002-93d1-7b2c08aadbb7\") " pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-tcl57"
Sep 11 22:31:07.213423 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:07.213294    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb8xh\" (UniqueName: \"kubernetes.io/projected/68a08d2e-2a17-4002-93d1-7b2c08aadbb7-kube-api-access-tb8xh\") pod \"maas-api-6945bc8b6b-tcl57\" (UID: \"68a08d2e-2a17-4002-93d1-7b2c08aadbb7\") " pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-tcl57"
Sep 11 22:31:07.214126 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:07.214098    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68a08d2e-2a17-4002-93d1-7b2c08aadbb7-openshift-service-ca\") pod \"maas-api-6945bc8b6b-tcl57\" (UID: \"68a08d2e-2a17-4002-93d1-7b2c08aadbb7\") " pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-tcl57"
Sep 11 22:31:07.215717 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:07.215689    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68a08d2e-2a17-4002-93d1-7b2c08aadbb7-maas-api-tls\") pod \"maas-api-6945bc8b6b-tcl57\" (UID: \"68a08d2e-2a17-4002-93d1-7b2c08aadbb7\") " pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-tcl57"
Sep 11 22:31:07.215873 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:07.215848    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68a08d2e-2a17-4002-93d1-7b2c08aadbb7-metrics-certs\") pod \"maas-api-6945bc8b6b-tcl57\" (UID: \"68a08d2e-2a17-4002-93d1-7b2c08aadbb7\") " pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-tcl57"
Sep 11 22:31:07.220920 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:07.220900    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb8xh\" (UniqueName: \"kubernetes.io/projected/68a08d2e-2a17-4002-93d1-7b2c08aadbb7-kube-api-access-tb8xh\") pod \"maas-api-6945bc8b6b-tcl57\" (UID: \"68a08d2e-2a17-4002-93d1-7b2c08aadbb7\") " pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-tcl57"
Sep 11 22:31:07.347085 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:07.347001    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-tcl57"
Sep 11 22:31:07.618510 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:07.618486    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6945bc8b6b-tcl57"]
Sep 11 22:31:07.620908 ip-10-0-139-88 kubenswrapper[2587]: W0911 22:31:07.620871    2587 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68a08d2e_2a17_4002_93d1_7b2c08aadbb7.slice/crio-34794ceb606312c8b2d5cca9b71a8108170e2447a62eef2a42236ae5ef94a2df WatchSource:0}: Error finding container 34794ceb606312c8b2d5cca9b71a8108170e2447a62eef2a42236ae5ef94a2df: Status 404 returned error can't find the container with id 34794ceb606312c8b2d5cca9b71a8108170e2447a62eef2a42236ae5ef94a2df
Sep 11 22:31:07.981624 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:07.981587    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-tcl57" event={"ID":"68a08d2e-2a17-4002-93d1-7b2c08aadbb7","Type":"ContainerStarted","Data":"34794ceb606312c8b2d5cca9b71a8108170e2447a62eef2a42236ae5ef94a2df"}
Sep 11 22:31:08.987873 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:08.987830    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-tcl57" event={"ID":"68a08d2e-2a17-4002-93d1-7b2c08aadbb7","Type":"ContainerStarted","Data":"a1be85fb155b3610e607608702f16fa0ef64f8224d5e916e3de4c5dc1ce82c2b"}
Sep 11 22:31:08.988298 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:08.987918    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-tcl57"
Sep 11 22:31:09.005420 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:09.005365    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-tcl57" podStartSLOduration=1.622018087 podStartE2EDuration="2.005300066s" totalImagesPullingTime="383.281979ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 22:31:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:31:09.004458414 +0000 UTC m=+2322.179534803" watchObservedRunningTime="2026-09-11 22:31:09.005300066 +0000 UTC m=+2322.180376436"
Sep 11 22:31:14.377449 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:14.377412    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:31:14.999060 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:14.999025    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6945bc8b6b-tcl57"
Sep 11 22:31:15.047581 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:15.047540    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-758d76856b-8dm4c"]
Sep 11 22:31:15.047828 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:15.047799    2587 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c" podUID="662be1ac-246e-43b9-a0b6-314e90293e1d" containerName="maas-api" containerID="cri-o://52b30a5db1a169418378dbf4915c0b36e483288c16e0d2a72c795cc0faa0f148" gracePeriod=30
Sep 11 22:31:15.856539 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:15.856500    2587 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c" podUID="662be1ac-246e-43b9-a0b6-314e90293e1d" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.42:8443/health\": dial tcp 10.134.0.42:8443: connect: connection refused"
Sep 11 22:31:16.300220 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:16.300197    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c"
Sep 11 22:31:16.397150 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:16.397051    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/662be1ac-246e-43b9-a0b6-314e90293e1d-openshift-service-ca\") pod \"662be1ac-246e-43b9-a0b6-314e90293e1d\" (UID: \"662be1ac-246e-43b9-a0b6-314e90293e1d\") "
Sep 11 22:31:16.397150 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:16.397107    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxpmj\" (UniqueName: \"kubernetes.io/projected/662be1ac-246e-43b9-a0b6-314e90293e1d-kube-api-access-cxpmj\") pod \"662be1ac-246e-43b9-a0b6-314e90293e1d\" (UID: \"662be1ac-246e-43b9-a0b6-314e90293e1d\") "
Sep 11 22:31:16.397425 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:16.397174    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/662be1ac-246e-43b9-a0b6-314e90293e1d-metrics-certs\") pod \"662be1ac-246e-43b9-a0b6-314e90293e1d\" (UID: \"662be1ac-246e-43b9-a0b6-314e90293e1d\") "
Sep 11 22:31:16.397425 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:16.397219    2587 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/662be1ac-246e-43b9-a0b6-314e90293e1d-maas-api-tls\") pod \"662be1ac-246e-43b9-a0b6-314e90293e1d\" (UID: \"662be1ac-246e-43b9-a0b6-314e90293e1d\") "
Sep 11 22:31:16.397561 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:16.397531    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/662be1ac-246e-43b9-a0b6-314e90293e1d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "662be1ac-246e-43b9-a0b6-314e90293e1d" (UID: "662be1ac-246e-43b9-a0b6-314e90293e1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 22:31:16.399369 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:16.399341    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/662be1ac-246e-43b9-a0b6-314e90293e1d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "662be1ac-246e-43b9-a0b6-314e90293e1d" (UID: "662be1ac-246e-43b9-a0b6-314e90293e1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:31:16.399499 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:16.399366    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/662be1ac-246e-43b9-a0b6-314e90293e1d-kube-api-access-cxpmj" (OuterVolumeSpecName: "kube-api-access-cxpmj") pod "662be1ac-246e-43b9-a0b6-314e90293e1d" (UID: "662be1ac-246e-43b9-a0b6-314e90293e1d"). InnerVolumeSpecName "kube-api-access-cxpmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 22:31:16.399499 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:16.399476    2587 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/662be1ac-246e-43b9-a0b6-314e90293e1d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "662be1ac-246e-43b9-a0b6-314e90293e1d" (UID: "662be1ac-246e-43b9-a0b6-314e90293e1d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 22:31:16.498389 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:16.498350    2587 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/662be1ac-246e-43b9-a0b6-314e90293e1d-maas-api-tls\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:31:16.498389 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:16.498382    2587 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/662be1ac-246e-43b9-a0b6-314e90293e1d-openshift-service-ca\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:31:16.498389 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:16.498395    2587 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxpmj\" (UniqueName: \"kubernetes.io/projected/662be1ac-246e-43b9-a0b6-314e90293e1d-kube-api-access-cxpmj\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:31:16.498633 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:16.498406    2587 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/662be1ac-246e-43b9-a0b6-314e90293e1d-metrics-certs\") on node \"ip-10-0-139-88.ec2.internal\" DevicePath \"\""
Sep 11 22:31:17.034642 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:17.034608    2587 generic.go:356] "Generic (PLEG): container finished" podID="662be1ac-246e-43b9-a0b6-314e90293e1d" containerID="52b30a5db1a169418378dbf4915c0b36e483288c16e0d2a72c795cc0faa0f148" exitCode=0
Sep 11 22:31:17.035036 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:17.034648    2587 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c"
Sep 11 22:31:17.035036 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:17.034667    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c" event={"ID":"662be1ac-246e-43b9-a0b6-314e90293e1d","Type":"ContainerDied","Data":"52b30a5db1a169418378dbf4915c0b36e483288c16e0d2a72c795cc0faa0f148"}
Sep 11 22:31:17.035036 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:17.034714    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-758d76856b-8dm4c" event={"ID":"662be1ac-246e-43b9-a0b6-314e90293e1d","Type":"ContainerDied","Data":"376d2a9408341ea4e9989399d88b61c95a12c0a8608b6c19b7dd12963c4e19ae"}
Sep 11 22:31:17.035036 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:17.034738    2587 scope.go:117] "RemoveContainer" containerID="52b30a5db1a169418378dbf4915c0b36e483288c16e0d2a72c795cc0faa0f148"
Sep 11 22:31:17.046396 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:17.046371    2587 scope.go:117] "RemoveContainer" containerID="52b30a5db1a169418378dbf4915c0b36e483288c16e0d2a72c795cc0faa0f148"
Sep 11 22:31:17.047227 ip-10-0-139-88 kubenswrapper[2587]: E0911 22:31:17.047200    2587 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52b30a5db1a169418378dbf4915c0b36e483288c16e0d2a72c795cc0faa0f148\": container with ID starting with 52b30a5db1a169418378dbf4915c0b36e483288c16e0d2a72c795cc0faa0f148 not found: ID does not exist" containerID="52b30a5db1a169418378dbf4915c0b36e483288c16e0d2a72c795cc0faa0f148"
Sep 11 22:31:17.047384 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:17.047358    2587 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52b30a5db1a169418378dbf4915c0b36e483288c16e0d2a72c795cc0faa0f148"} err="failed to get container status \"52b30a5db1a169418378dbf4915c0b36e483288c16e0d2a72c795cc0faa0f148\": rpc error: code = NotFound desc = could not find container \"52b30a5db1a169418378dbf4915c0b36e483288c16e0d2a72c795cc0faa0f148\": container with ID starting with 52b30a5db1a169418378dbf4915c0b36e483288c16e0d2a72c795cc0faa0f148 not found: ID does not exist"
Sep 11 22:31:17.062027 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:17.061999    2587 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-758d76856b-8dm4c"]
Sep 11 22:31:17.064014 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:17.063990    2587 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-758d76856b-8dm4c"]
Sep 11 22:31:17.337669 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:17.337590    2587 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="662be1ac-246e-43b9-a0b6-314e90293e1d" path="/var/lib/kubelet/pods/662be1ac-246e-43b9-a0b6-314e90293e1d/volumes"
Sep 11 22:31:21.772523 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:21.772442    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:31:29.670317 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:29.670277    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:31:34.670283 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:34.670247    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tw9k5"]
Sep 11 22:31:37.847297 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:37.847256    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-797db855dd-vt4lr_30640515-d446-40a7-b8bb-f8aa52b448ba/authorino/0.log"
Sep 11 22:31:43.275903 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:43.275862    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-rlw6w_c6895de2-3e55-4c67-8107-7ed453c7a934/manager/0.log"
Sep 11 22:31:43.399899 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:43.399867    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-lkxdk_c57c2e4e-3ab4-4276-9016-b9ff4a0202fc/manager/0.log"
Sep 11 22:31:43.531046 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:43.530956    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fcd74f577-5hnlp_515308b9-9cbd-4828-aee1-48f55f701bac/manager/0.log"
Sep 11 22:31:43.893106 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:43.893017    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bf8845876-7pms6_b88b394c-61ba-4af9-b3d3-fb5ef027cefe/manager/0.log"
Sep 11 22:31:43.898998 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:43.898973    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bf8845876-7pms6_b88b394c-61ba-4af9-b3d3-fb5ef027cefe/copy-manifests/0.log"
Sep 11 22:31:44.623758 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:44.623731    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6945bc8b6b-tcl57_68a08d2e-2a17-4002-93d1-7b2c08aadbb7/maas-api/0.log"
Sep 11 22:31:45.768709 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:45.768676    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-797db855dd-vt4lr_30640515-d446-40a7-b8bb-f8aa52b448ba/authorino/0.log"
Sep 11 22:31:45.890252 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:45.890220    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-mcqjh_096c0792-1d86-45fc-a9e3-ecfc43321e48/manager/0.log"
Sep 11 22:31:46.001205 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:46.001152    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-rcj2k_77cee7bd-677f-4371-982a-49c059de9b54/manager/0.log"
Sep 11 22:31:46.114932 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:46.114857    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-64t4p_06b4da6f-ba91-40be-8cb9-12c20797c691/kuadrant-console-plugin/0.log"
Sep 11 22:31:46.366021 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:46.365930    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-tw9k5_13759088-b5e2-4daa-8795-496e47f511eb/limitador/0.log"
Sep 11 22:31:46.490685 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:46.490655    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-vjrz7_851204ff-15b4-40e8-9558-1d6fd04cfabf/manager/0.log"
Sep 11 22:31:47.207805 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:47.207775    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-576b4b4bb7-v29kw_a979a0c2-3065-4255-bae3-458f48999604/kube-auth-proxy/0.log"
Sep 11 22:31:47.330483 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:47.330447    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-jtsp7_60203398-7954-425d-8699-a6c2bfd7bddb/istio-proxy/1.log"
Sep 11 22:31:47.576388 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:47.576288    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-bsmwj_68c8f814-bf9c-4c1e-a642-741074aef1fd/payload-processing/0.log"
Sep 11 22:31:48.504870 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:48.504835    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fjzqcg_4ef083be-b3c6-41b8-b844-213b2fd574db/main/0.log"
Sep 11 22:31:57.703755 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:57.703721    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fc5j4_05523d8a-2c9a-47f3-ad9d-4c508546cd42/global-pull-secret-syncer/0.log"
Sep 11 22:31:57.797982 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:57.797944    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9j5p9_fb56d2f4-9566-4726-81dd-39cf43dbbc42/konnectivity-agent/0.log"
Sep 11 22:31:57.887709 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:31:57.887658    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-88.ec2.internal_d56814d1b26653aa054a4eb19dfe6290/haproxy/0.log"
Sep 11 22:32:02.539818 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:02.539763    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-797db855dd-vt4lr_30640515-d446-40a7-b8bb-f8aa52b448ba/authorino/0.log"
Sep 11 22:32:02.617661 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:02.617624    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-mcqjh_096c0792-1d86-45fc-a9e3-ecfc43321e48/manager/0.log"
Sep 11 22:32:02.639342 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:02.639303    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-rcj2k_77cee7bd-677f-4371-982a-49c059de9b54/manager/0.log"
Sep 11 22:32:02.663597 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:02.663570    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-64t4p_06b4da6f-ba91-40be-8cb9-12c20797c691/kuadrant-console-plugin/0.log"
Sep 11 22:32:02.742886 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:02.742848    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-tw9k5_13759088-b5e2-4daa-8795-496e47f511eb/limitador/0.log"
Sep 11 22:32:02.822615 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:02.822525    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-vjrz7_851204ff-15b4-40e8-9558-1d6fd04cfabf/manager/0.log"
Sep 11 22:32:04.812880 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:04.812850    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5b7987cc77-2xgzc_9380b76d-3c91-414b-a9db-5ec96971b603/metrics-server/0.log"
Sep 11 22:32:04.883530 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:04.883442    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8zz2r_194dd741-32d8-4ff6-be38-a16c389965fb/node-exporter/0.log"
Sep 11 22:32:04.903542 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:04.903508    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8zz2r_194dd741-32d8-4ff6-be38-a16c389965fb/kube-rbac-proxy/0.log"
Sep 11 22:32:04.927036 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:04.927006    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8zz2r_194dd741-32d8-4ff6-be38-a16c389965fb/init-textfile/0.log"
Sep 11 22:32:05.174270 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:05.174233    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-fbsjn_f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6/kube-rbac-proxy-main/0.log"
Sep 11 22:32:05.195687 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:05.195661    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-fbsjn_f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6/kube-rbac-proxy-self/0.log"
Sep 11 22:32:05.225606 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:05.225579    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-fbsjn_f2fe79e0-afd9-4e59-91f8-76a8d0bbc2a6/openshift-state-metrics/0.log"
Sep 11 22:32:05.553917 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:05.553840    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5cf76b897f-jbmhr_c3de3479-780f-4ea1-9eb4-8cc83cfa8deb/telemeter-client/0.log"
Sep 11 22:32:05.579686 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:05.579663    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5cf76b897f-jbmhr_c3de3479-780f-4ea1-9eb4-8cc83cfa8deb/reload/0.log"
Sep 11 22:32:05.601135 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:05.601102    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5cf76b897f-jbmhr_c3de3479-780f-4ea1-9eb4-8cc83cfa8deb/kube-rbac-proxy/0.log"
Sep 11 22:32:07.161378 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:07.161343    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-wplsx_d10cc55c-a4d2-4c30-84d9-15ced74f00b3/networking-console-plugin/0.log"
Sep 11 22:32:07.763149 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:07.763117    2587 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vbnb9/perf-node-gather-daemonset-8vh4b"]
Sep 11 22:32:07.763537 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:07.763516    2587 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="662be1ac-246e-43b9-a0b6-314e90293e1d" containerName="maas-api"
Sep 11 22:32:07.763596 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:07.763538    2587 state_mem.go:107] "Deleted CPUSet assignment" podUID="662be1ac-246e-43b9-a0b6-314e90293e1d" containerName="maas-api"
Sep 11 22:32:07.763633 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:07.763612    2587 memory_manager.go:368] "RemoveStaleState removing state" podUID="662be1ac-246e-43b9-a0b6-314e90293e1d" containerName="maas-api"
Sep 11 22:32:07.766652 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:07.766632    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-8vh4b"
Sep 11 22:32:07.769163 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:07.769138    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vbnb9\"/\"kube-root-ca.crt\""
Sep 11 22:32:07.769543 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:07.769524    2587 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vbnb9\"/\"openshift-service-ca.crt\""
Sep 11 22:32:07.770503 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:07.770488    2587 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vbnb9\"/\"default-dockercfg-pz8hq\""
Sep 11 22:32:07.776843 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:07.776819    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vbnb9/perf-node-gather-daemonset-8vh4b"]
Sep 11 22:32:07.877487 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:07.877455    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/02ecc669-fbb4-48b6-ad68-ce931360ca46-proc\") pod \"perf-node-gather-daemonset-8vh4b\" (UID: \"02ecc669-fbb4-48b6-ad68-ce931360ca46\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-8vh4b"
Sep 11 22:32:07.877669 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:07.877502    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfw6p\" (UniqueName: \"kubernetes.io/projected/02ecc669-fbb4-48b6-ad68-ce931360ca46-kube-api-access-cfw6p\") pod \"perf-node-gather-daemonset-8vh4b\" (UID: \"02ecc669-fbb4-48b6-ad68-ce931360ca46\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-8vh4b"
Sep 11 22:32:07.877669 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:07.877591    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02ecc669-fbb4-48b6-ad68-ce931360ca46-lib-modules\") pod \"perf-node-gather-daemonset-8vh4b\" (UID: \"02ecc669-fbb4-48b6-ad68-ce931360ca46\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-8vh4b"
Sep 11 22:32:07.877756 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:07.877672    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02ecc669-fbb4-48b6-ad68-ce931360ca46-sys\") pod \"perf-node-gather-daemonset-8vh4b\" (UID: \"02ecc669-fbb4-48b6-ad68-ce931360ca46\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-8vh4b"
Sep 11 22:32:07.877756 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:07.877695    2587 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/02ecc669-fbb4-48b6-ad68-ce931360ca46-podres\") pod \"perf-node-gather-daemonset-8vh4b\" (UID: \"02ecc669-fbb4-48b6-ad68-ce931360ca46\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-8vh4b"
Sep 11 22:32:07.978827 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:07.978789    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02ecc669-fbb4-48b6-ad68-ce931360ca46-sys\") pod \"perf-node-gather-daemonset-8vh4b\" (UID: \"02ecc669-fbb4-48b6-ad68-ce931360ca46\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-8vh4b"
Sep 11 22:32:07.978827 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:07.978828    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/02ecc669-fbb4-48b6-ad68-ce931360ca46-podres\") pod \"perf-node-gather-daemonset-8vh4b\" (UID: \"02ecc669-fbb4-48b6-ad68-ce931360ca46\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-8vh4b"
Sep 11 22:32:07.979045 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:07.978876    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/02ecc669-fbb4-48b6-ad68-ce931360ca46-proc\") pod \"perf-node-gather-daemonset-8vh4b\" (UID: \"02ecc669-fbb4-48b6-ad68-ce931360ca46\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-8vh4b"
Sep 11 22:32:07.979045 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:07.978899    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfw6p\" (UniqueName: \"kubernetes.io/projected/02ecc669-fbb4-48b6-ad68-ce931360ca46-kube-api-access-cfw6p\") pod \"perf-node-gather-daemonset-8vh4b\" (UID: \"02ecc669-fbb4-48b6-ad68-ce931360ca46\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-8vh4b"
Sep 11 22:32:07.979045 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:07.978930    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02ecc669-fbb4-48b6-ad68-ce931360ca46-sys\") pod \"perf-node-gather-daemonset-8vh4b\" (UID: \"02ecc669-fbb4-48b6-ad68-ce931360ca46\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-8vh4b"
Sep 11 22:32:07.979045 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:07.978935    2587 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02ecc669-fbb4-48b6-ad68-ce931360ca46-lib-modules\") pod \"perf-node-gather-daemonset-8vh4b\" (UID: \"02ecc669-fbb4-48b6-ad68-ce931360ca46\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-8vh4b"
Sep 11 22:32:07.979045 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:07.978988    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/02ecc669-fbb4-48b6-ad68-ce931360ca46-proc\") pod \"perf-node-gather-daemonset-8vh4b\" (UID: \"02ecc669-fbb4-48b6-ad68-ce931360ca46\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-8vh4b"
Sep 11 22:32:07.979232 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:07.979048    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02ecc669-fbb4-48b6-ad68-ce931360ca46-lib-modules\") pod \"perf-node-gather-daemonset-8vh4b\" (UID: \"02ecc669-fbb4-48b6-ad68-ce931360ca46\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-8vh4b"
Sep 11 22:32:07.979232 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:07.979048    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/02ecc669-fbb4-48b6-ad68-ce931360ca46-podres\") pod \"perf-node-gather-daemonset-8vh4b\" (UID: \"02ecc669-fbb4-48b6-ad68-ce931360ca46\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-8vh4b"
Sep 11 22:32:07.986809 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:07.986778    2587 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfw6p\" (UniqueName: \"kubernetes.io/projected/02ecc669-fbb4-48b6-ad68-ce931360ca46-kube-api-access-cfw6p\") pod \"perf-node-gather-daemonset-8vh4b\" (UID: \"02ecc669-fbb4-48b6-ad68-ce931360ca46\") " pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-8vh4b"
Sep 11 22:32:08.077256 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:08.077165    2587 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-8vh4b"
Sep 11 22:32:08.350491 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:08.350468    2587 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vbnb9/perf-node-gather-daemonset-8vh4b"]
Sep 11 22:32:08.384798 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:08.384771    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5d9fbf6d78-f9vx9_e4d1f5cc-f268-46f6-8cce-b126370c7def/console/0.log"
Sep 11 22:32:08.436945 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:08.436920    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-6p4cp_8c10e945-fa90-4b6a-b5a8-423c05d6db4d/download-server/0.log"
Sep 11 22:32:09.292289 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:09.292248    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-8vh4b" event={"ID":"02ecc669-fbb4-48b6-ad68-ce931360ca46","Type":"ContainerStarted","Data":"764dd5d0cc84fb32c77c8c47ebe38c929cc9d2432f69f5b3ba92a3dca59f840c"}
Sep 11 22:32:09.292530 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:09.292302    2587 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-8vh4b" event={"ID":"02ecc669-fbb4-48b6-ad68-ce931360ca46","Type":"ContainerStarted","Data":"bde9b7f8028978392966d0a45e00a741082c612f4c9a421887b0dda5a8eb01f9"}
Sep 11 22:32:09.292530 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:09.292429    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-8vh4b"
Sep 11 22:32:09.309093 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:09.309044    2587 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-8vh4b" podStartSLOduration=2.309031025 podStartE2EDuration="2.309031025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 22:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 22:32:09.30703509 +0000 UTC m=+2382.482111481" watchObservedRunningTime="2026-09-11 22:32:09.309031025 +0000 UTC m=+2382.484107429"
Sep 11 22:32:10.337116 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:10.337082    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rn9mr_10261a6e-338b-4c67-99bd-6921a24e051b/dns/0.log"
Sep 11 22:32:10.356389 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:10.356360    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rn9mr_10261a6e-338b-4c67-99bd-6921a24e051b/kube-rbac-proxy/0.log"
Sep 11 22:32:10.379501 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:10.379473    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-756n4_50fccc19-46f5-4bdc-8e4d-c728752de971/dns-node-resolver/0.log"
Sep 11 22:32:11.000744 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:11.000709    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-56b94bddb8-spm56_159a6253-516b-4444-81d9-d7daa7b247af/registry/0.log"
Sep 11 22:32:11.024780 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:11.024738    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4vqqq_87753932-baf9-409d-948e-3d9eb12b1095/node-ca/0.log"
Sep 11 22:32:12.558069 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:12.558035    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-576b4b4bb7-v29kw_a979a0c2-3065-4255-bae3-458f48999604/kube-auth-proxy/0.log"
Sep 11 22:32:12.585152 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:12.585115    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-jtsp7_60203398-7954-425d-8699-a6c2bfd7bddb/istio-proxy/0.log"
Sep 11 22:32:12.589755 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:12.589729    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-jtsp7_60203398-7954-425d-8699-a6c2bfd7bddb/istio-proxy/1.log"
Sep 11 22:32:12.659954 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:12.659920    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-bsmwj_68c8f814-bf9c-4c1e-a642-741074aef1fd/payload-processing/0.log"
Sep 11 22:32:13.634047 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:13.634012    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dnlj8_3cdabfb8-9e10-4fe2-9ba4-4301f2ea9fd0/serve-healthcheck-canary/0.log"
Sep 11 22:32:14.293274 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:14.293243    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c9qjm_3fdc3282-48a9-49ae-a5a4-5d60fb93639a/kube-rbac-proxy/0.log"
Sep 11 22:32:14.313263 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:14.313235    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c9qjm_3fdc3282-48a9-49ae-a5a4-5d60fb93639a/exporter/0.log"
Sep 11 22:32:14.333470 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:14.333436    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c9qjm_3fdc3282-48a9-49ae-a5a4-5d60fb93639a/extractor/0.log"
Sep 11 22:32:15.307777 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:15.307751    2587 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vbnb9/perf-node-gather-daemonset-8vh4b"
Sep 11 22:32:16.536564 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:16.536530    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-rlw6w_c6895de2-3e55-4c67-8107-7ed453c7a934/manager/0.log"
Sep 11 22:32:17.035129 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:17.035038    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-lkxdk_c57c2e4e-3ab4-4276-9016-b9ff4a0202fc/manager/0.log"
Sep 11 22:32:17.105277 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:17.105240    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fcd74f577-5hnlp_515308b9-9cbd-4828-aee1-48f55f701bac/manager/0.log"
Sep 11 22:32:17.338268 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:17.338183    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bf8845876-7pms6_b88b394c-61ba-4af9-b3d3-fb5ef027cefe/manager/0.log"
Sep 11 22:32:17.357615 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:17.357585    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5bf8845876-7pms6_b88b394c-61ba-4af9-b3d3-fb5ef027cefe/copy-manifests/0.log"
Sep 11 22:32:19.114844 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:19.114811    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-576b4b4bb7-v29kw_a979a0c2-3065-4255-bae3-458f48999604/kube-auth-proxy/0.log"
Sep 11 22:32:19.137939 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:19.137895    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-jtsp7_60203398-7954-425d-8699-a6c2bfd7bddb/istio-proxy/0.log"
Sep 11 22:32:19.142920 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:19.142892    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-jtsp7_60203398-7954-425d-8699-a6c2bfd7bddb/istio-proxy/1.log"
Sep 11 22:32:19.199514 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:19.199414    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-bsmwj_68c8f814-bf9c-4c1e-a642-741074aef1fd/payload-processing/0.log"
Sep 11 22:32:21.208169 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:21.208131    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-576b4b4bb7-v29kw_a979a0c2-3065-4255-bae3-458f48999604/kube-auth-proxy/0.log"
Sep 11 22:32:21.234944 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:21.234910    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-jtsp7_60203398-7954-425d-8699-a6c2bfd7bddb/istio-proxy/0.log"
Sep 11 22:32:21.239997 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:21.239967    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-jtsp7_60203398-7954-425d-8699-a6c2bfd7bddb/istio-proxy/1.log"
Sep 11 22:32:21.297467 ip-10-0-139-88 kubenswrapper[2587]: I0911 22:32:21.297428    2587 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-bsmwj_68c8f814-bf9c-4c1e-a642-741074aef1fd/payload-processing/0.log"