Sep 16 12:01:18.856447 ip-10-0-137-30 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 12:01:18.856459 ip-10-0-137-30 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 12:01:18.856469 ip-10-0-137-30 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 12:01:18.856676 ip-10-0-137-30 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 12:01:30.145173 ip-10-0-137-30 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 12:01:30.145209 ip-10-0-137-30 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e023b5acceb647748a62609b53479e2c --
Sep 16 12:03:26.186948 ip-10-0-137-30 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 12:03:26.572168 ip-10-0-137-30 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:03:26.572168 ip-10-0-137-30 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 12:03:26.572168 ip-10-0-137-30 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:03:26.572168 ip-10-0-137-30 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 12:03:26.572168 ip-10-0-137-30 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 12:03:26.573086 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.573000    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 12:03:26.575930 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.575910    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:03:26.575930 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.575928    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:03:26.575930 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.575932    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:03:26.575930 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.575935    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:03:26.576079 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.575938    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:03:26.576079 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.575942    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:03:26.576079 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.575944    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:03:26.576079 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.575947    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:03:26.576079 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.575950    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:03:26.576079 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.575952    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:03:26.576079 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.575955    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:03:26.576079 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.575957    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:03:26.576079 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.575960    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:03:26.576079 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.575963    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:03:26.576079 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.575965    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:03:26.576079 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.575968    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:03:26.576079 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.575971    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:03:26.576079 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.575973    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:03:26.576079 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.575982    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:03:26.576079 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.575985    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:03:26.576079 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.575988    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:03:26.576079 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.575990    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:03:26.576079 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.575993    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:03:26.576079 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.575996    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:03:26.576551 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.575998    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:03:26.576551 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576001    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:03:26.576551 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576003    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:03:26.576551 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576006    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:03:26.576551 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576008    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:03:26.576551 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576011    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:03:26.576551 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576013    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:03:26.576551 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576016    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:03:26.576551 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576019    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:03:26.576551 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576021    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:03:26.576551 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576023    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:03:26.576551 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576026    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:03:26.576551 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576029    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:03:26.576551 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576032    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:03:26.576551 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576035    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:03:26.576551 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576038    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:03:26.576551 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576040    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:03:26.576551 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576043    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:03:26.576551 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576045    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:03:26.576551 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576048    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:03:26.577058 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576050    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:03:26.577058 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576053    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:03:26.577058 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576055    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:03:26.577058 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576058    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:03:26.577058 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576060    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:03:26.577058 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576063    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:03:26.577058 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576065    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:03:26.577058 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576069    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:03:26.577058 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576071    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:03:26.577058 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576074    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:03:26.577058 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576076    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:03:26.577058 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576079    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:03:26.577058 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576082    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:03:26.577058 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576085    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:03:26.577058 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576088    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:03:26.577058 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576091    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:03:26.577058 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576093    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:03:26.577058 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576096    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:03:26.577058 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576098    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:03:26.577058 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576101    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:03:26.577539 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576103    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:03:26.577539 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576106    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:03:26.577539 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576108    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:03:26.577539 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576111    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:03:26.577539 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576114    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:03:26.577539 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576117    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:03:26.577539 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576120    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:03:26.577539 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576122    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:03:26.577539 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576126    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:03:26.577539 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576130    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:03:26.577539 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576133    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:03:26.577539 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576137    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:03:26.577539 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576140    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:03:26.577539 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576143    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:03:26.577539 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576146    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:03:26.577539 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576149    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:03:26.577539 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576152    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:03:26.577539 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576155    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:03:26.577539 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576157    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:03:26.578057 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576160    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:03:26.578057 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576163    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:03:26.578057 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576166    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:03:26.578057 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576170    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:03:26.578057 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576173    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:03:26.578057 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576176    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:03:26.578057 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576178    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:03:26.578057 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576181    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:03:26.578057 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576183    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:03:26.578057 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576186    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:03:26.578057 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576189    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:03:26.578057 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576191    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:03:26.578057 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576194    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:03:26.578057 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576196    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:03:26.578057 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576199    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:03:26.578057 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576202    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:03:26.578057 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576894    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:03:26.578057 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576901    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:03:26.578057 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576906    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:03:26.578057 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576909    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:03:26.578537 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576912    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:03:26.578537 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576915    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:03:26.578537 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576917    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:03:26.578537 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576920    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:03:26.578537 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576923    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:03:26.578537 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576927    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:03:26.578537 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576931    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:03:26.578537 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576934    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:03:26.578537 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576936    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:03:26.578537 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576939    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:03:26.578537 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576942    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:03:26.578537 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576944    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:03:26.578537 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576947    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:03:26.578537 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576949    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:03:26.578537 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576952    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:03:26.578537 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576954    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:03:26.578537 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576957    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:03:26.578537 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576960    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:03:26.578537 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576963    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:03:26.578537 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576965    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:03:26.579062 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576968    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:03:26.579062 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576970    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:03:26.579062 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576973    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:03:26.579062 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576975    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:03:26.579062 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576978    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:03:26.579062 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576980    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:03:26.579062 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576983    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:03:26.579062 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576985    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:03:26.579062 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576988    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:03:26.579062 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576992    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:03:26.579062 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576996    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:03:26.579062 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.576999    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:03:26.579062 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577003    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:03:26.579062 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577006    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:03:26.579062 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577009    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:03:26.579062 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577011    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:03:26.579062 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577014    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:03:26.579062 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577016    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:03:26.579062 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577019    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:03:26.579532 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577022    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:03:26.579532 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577024    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:03:26.579532 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577027    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:03:26.579532 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577029    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:03:26.579532 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577032    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:03:26.579532 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577034    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:03:26.579532 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577037    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:03:26.579532 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577039    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:03:26.579532 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577042    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:03:26.579532 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577044    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:03:26.579532 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577047    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:03:26.579532 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577049    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:03:26.579532 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577052    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:03:26.579532 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577054    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:03:26.579532 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577057    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:03:26.579532 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577059    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:03:26.579532 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577063    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:03:26.579532 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577066    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:03:26.579532 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577068    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:03:26.579532 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577071    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:03:26.580043 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577074    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:03:26.580043 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577076    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:03:26.580043 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577079    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:03:26.580043 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577082    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:03:26.580043 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577084    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:03:26.580043 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577087    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:03:26.580043 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577090    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:03:26.580043 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577094    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:03:26.580043 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577097    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:03:26.580043 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577100    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:03:26.580043 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577103    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:03:26.580043 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577106    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:03:26.580043 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577109    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:03:26.580043 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577112    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:03:26.580043 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577115    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:03:26.580043 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577118    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:03:26.580043 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577120    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:03:26.580043 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577123    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:03:26.580043 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577127    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:03:26.580518 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577130    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:03:26.580518 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577133    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:03:26.580518 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577136    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:03:26.580518 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577139    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:03:26.580518 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577141    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:03:26.580518 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577144    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:03:26.580518 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577147    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:03:26.580518 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577150    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:03:26.580518 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577152    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:03:26.580518 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577155    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:03:26.580518 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577158    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:03:26.580518 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577160    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:03:26.580518 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577163    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:03:26.580518 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577166    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:03:26.580518 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577169    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:03:26.580518 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577171    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:03:26.580518 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577174    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:03:26.580518 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577587    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:03:26.580518 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577593    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:03:26.580518 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577600    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:03:26.581031 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577605    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:03:26.581031 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577609    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:03:26.581031 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577612    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:03:26.581031 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577615    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:03:26.581031 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577618    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:03:26.581031 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577620    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:03:26.581031 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577623    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:03:26.581031 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577626    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:03:26.581031 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577628    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:03:26.581031 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577631    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:03:26.581031 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577633    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:03:26.581031 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577636    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:03:26.581031 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577638    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:03:26.581031 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577641    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:03:26.581031 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577644    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:03:26.581031 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577646    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:03:26.581031 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577649    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:03:26.581031 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577668    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:03:26.581031 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577671    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:03:26.581497 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577674    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:03:26.581497 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577677    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:03:26.581497 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577680    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:03:26.581497 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577683    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:03:26.581497 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577685    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:03:26.581497 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577688    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:03:26.581497 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577690    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:03:26.581497 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577693    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:03:26.581497 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577695    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:03:26.581497 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577699    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:03:26.581497 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577701    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:03:26.581497 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577704    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:03:26.581497 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577708    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:03:26.581497 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577712    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:03:26.581497 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577714    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:03:26.581497 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577718    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:03:26.581497 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577721    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:03:26.581497 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577725    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:03:26.581497 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577727    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:03:26.581985 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577730    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:03:26.581985 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577733    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:03:26.581985 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577736    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:03:26.581985 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577738    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:03:26.581985 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577741    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:03:26.581985 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577744    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:03:26.581985 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577747    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:03:26.581985 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577749    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:03:26.581985 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577752    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:03:26.581985 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577755    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:03:26.581985 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577759    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:03:26.581985 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577762    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:03:26.581985 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577765    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:03:26.581985 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577768    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:03:26.581985 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577770    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:03:26.581985 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577772    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:03:26.581985 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577775    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:03:26.581985 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577777    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:03:26.581985 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577780    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:03:26.581985 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577782    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:03:26.582471 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577785    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:03:26.582471 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577788    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:03:26.582471 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577790    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:03:26.582471 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577793    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:03:26.582471 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577797    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:03:26.582471 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577800    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:03:26.582471 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577803    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:03:26.582471 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577806    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:03:26.582471 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577809    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:03:26.582471 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577811    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:03:26.582471 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577814    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:03:26.582471 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577816    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:03:26.582471 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577819    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:03:26.582471 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577821    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:03:26.582471 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577824    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:03:26.582471 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577826    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:03:26.582471 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577828    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:03:26.582471 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577831    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:03:26.582471 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577833    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:03:26.582986 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577836    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:03:26.582986 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577838    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:03:26.582986 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577840    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:03:26.582986 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577843    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:03:26.582986 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577846    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:03:26.582986 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577848    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:03:26.582986 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577851    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:03:26.582986 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577853    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:03:26.582986 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577855    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:03:26.582986 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577858    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:03:26.582986 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577860    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:03:26.582986 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577863    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:03:26.582986 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577865    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:03:26.582986 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577867    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:03:26.582986 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577870    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:03:26.582986 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577873    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:03:26.582986 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577875    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:03:26.582986 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577878    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:03:26.582986 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.577882    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:03:26.582986 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.577949    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 12:03:26.582986 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.577982    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 12:03:26.583507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.577991    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 12:03:26.583507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.577996    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 12:03:26.583507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578000    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 12:03:26.583507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578003    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 12:03:26.583507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578007    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 12:03:26.583507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578011    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 12:03:26.583507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578014    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 12:03:26.583507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578017    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 12:03:26.583507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578021    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 12:03:26.583507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578024    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 12:03:26.583507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578027    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 12:03:26.583507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578030    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 16 12:03:26.583507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578033    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 12:03:26.583507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578036    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 16 12:03:26.583507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578039    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 12:03:26.583507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578042    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 12:03:26.583507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578046    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 16 12:03:26.583507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578049    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 12:03:26.583507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578052    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 12:03:26.583507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578056    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 12:03:26.583507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578059    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 12:03:26.583507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578063    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 12:03:26.583507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578066    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 12:03:26.584074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578070    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 12:03:26.584074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578073    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 12:03:26.584074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578076    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 12:03:26.584074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578079    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 12:03:26.584074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578082    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 12:03:26.584074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578085    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 12:03:26.584074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578089    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 12:03:26.584074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578093    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 12:03:26.584074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578096    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 12:03:26.584074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578099    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 12:03:26.584074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578102    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 12:03:26.584074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578105    2579 flags.go:64] FLAG: --enable-server="true"
Sep 16 12:03:26.584074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578108    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 12:03:26.584074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578112    2579 flags.go:64] FLAG: --event-burst="100"
Sep 16 12:03:26.584074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578115    2579 flags.go:64] FLAG: --event-qps="50"
Sep 16 12:03:26.584074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578117    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 12:03:26.584074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578120    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 12:03:26.584074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578123    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 16 12:03:26.584074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578127    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 12:03:26.584074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578130    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 12:03:26.584074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578132    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 12:03:26.584074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578136    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 16 12:03:26.584074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578139    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 12:03:26.584074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578141    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 12:03:26.584074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578144    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 12:03:26.584705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578147    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 12:03:26.584705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578150    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 12:03:26.584705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578153    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 12:03:26.584705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578156    2579 flags.go:64] FLAG: --feature-gates=""
Sep 16 12:03:26.584705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578160    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 12:03:26.584705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578164    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 12:03:26.584705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578167    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 12:03:26.584705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578171    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 12:03:26.584705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578173    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 12:03:26.584705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578177    2579 flags.go:64] FLAG: --help="false"
Sep 16 12:03:26.584705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578179    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-137-30.ec2.internal"
Sep 16 12:03:26.584705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578183    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 12:03:26.584705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578186    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 12:03:26.584705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578189    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 12:03:26.584705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578192    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 12:03:26.584705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578196    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 12:03:26.584705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578199    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 12:03:26.584705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578202    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 12:03:26.584705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578205    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 12:03:26.584705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578208    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 12:03:26.584705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578211    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 12:03:26.584705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578214    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 12:03:26.584705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578217    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 16 12:03:26.584705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578220    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 12:03:26.585307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578223    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 12:03:26.585307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578226    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 12:03:26.585307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578229    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 12:03:26.585307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578232    2579 flags.go:64] FLAG: --lock-file=""
Sep 16 12:03:26.585307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578235    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 12:03:26.585307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578238    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 12:03:26.585307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578241    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 12:03:26.585307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578246    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 12:03:26.585307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578249    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 12:03:26.585307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578252    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 12:03:26.585307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578255    2579 flags.go:64] FLAG: --logging-format="text"
Sep 16 12:03:26.585307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578258    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 12:03:26.585307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578261    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 12:03:26.585307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578265    2579 flags.go:64] FLAG: --manifest-url=""
Sep 16 12:03:26.585307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578268    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 12:03:26.585307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578273    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 12:03:26.585307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578276    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 12:03:26.585307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578280    2579 flags.go:64] FLAG: --max-pods="110"
Sep 16 12:03:26.585307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578283    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 12:03:26.585307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578286    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 12:03:26.585307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578288    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 12:03:26.585307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578291    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 12:03:26.585307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578294    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 12:03:26.585307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578297    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 12:03:26.585307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578300    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 12:03:26.585977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578307    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 12:03:26.585977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578310    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 12:03:26.585977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578314    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 12:03:26.585977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578317    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 16 12:03:26.585977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578320    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 12:03:26.585977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578326    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 12:03:26.585977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578328    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 12:03:26.585977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578332    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 12:03:26.585977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578335    2579 flags.go:64] FLAG: --port="10250"
Sep 16 12:03:26.585977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578338    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 12:03:26.585977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578341    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0835770f5f9c5803b"
Sep 16 12:03:26.585977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578344    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 16 12:03:26.585977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578347    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 12:03:26.585977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578350    2579 flags.go:64] FLAG: --register-node="true"
Sep 16 12:03:26.585977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578353    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 16 12:03:26.585977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578357    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 16 12:03:26.585977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578359    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 16 12:03:26.585977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578362    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 12:03:26.585977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578365    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 16 12:03:26.585977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578369    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 12:03:26.585977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578372    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 12:03:26.585977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578376    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 12:03:26.585977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578379    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 12:03:26.585977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578382    2579 flags.go:64] FLAG: --runonce="false"
Sep 16 12:03:26.585977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578385    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 12:03:26.586604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578388    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 12:03:26.586604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578391    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 12:03:26.586604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578394    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 12:03:26.586604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578397    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 12:03:26.586604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578400    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 12:03:26.586604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578403    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 12:03:26.586604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578406    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 12:03:26.586604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578409    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 12:03:26.586604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578412    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 12:03:26.586604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578416    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 12:03:26.586604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578421    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 12:03:26.586604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578427    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 12:03:26.586604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578430    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 16 12:03:26.586604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578433    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 12:03:26.586604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578438    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 12:03:26.586604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578441    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 12:03:26.586604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578444    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 12:03:26.586604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578448    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 16 12:03:26.586604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578451    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 12:03:26.586604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578454    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 12:03:26.586604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578457    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 12:03:26.586604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578461    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 12:03:26.586604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578464    2579 flags.go:64] FLAG: --v="2"
Sep 16 12:03:26.586604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578468    2579 flags.go:64] FLAG: --version="false"
Sep 16 12:03:26.586604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578472    2579 flags.go:64] FLAG: --vmodule=""
Sep 16 12:03:26.587229 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578476    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 12:03:26.587229 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.578479    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 12:03:26.587229 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578590    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:03:26.587229 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578595    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:03:26.587229 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578598    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:03:26.587229 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578601    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:03:26.587229 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578603    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:03:26.587229 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578607    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:03:26.587229 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578610    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:03:26.587229 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578613    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:03:26.587229 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578616    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:03:26.587229 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578619    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:03:26.587229 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578622    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:03:26.587229 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578625    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:03:26.587229 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578627    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:03:26.587229 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578630    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:03:26.587229 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578632    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:03:26.587229 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578635    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:03:26.587229 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578638    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:03:26.587229 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578641    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:03:26.587766 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578643    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:03:26.587766 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578646    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:03:26.587766 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578648    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:03:26.587766 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578651    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:03:26.587766 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578668    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:03:26.587766 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578670    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:03:26.587766 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578673    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:03:26.587766 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578676    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:03:26.587766 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578681    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:03:26.587766 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578683    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:03:26.587766 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578686    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:03:26.587766 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578689    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:03:26.587766 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578691    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:03:26.587766 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578694    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:03:26.587766 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578698    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:03:26.587766 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578703    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:03:26.587766 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578708    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:03:26.587766 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578711    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:03:26.587766 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578713    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:03:26.588498 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578716    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:03:26.588498 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578719    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:03:26.588498 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578721    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:03:26.588498 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578724    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:03:26.588498 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578727    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:03:26.588498 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578729    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:03:26.588498 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578732    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:03:26.588498 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578735    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:03:26.588498 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578738    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:03:26.588498 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578740    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:03:26.588498 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578743    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:03:26.588498 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578746    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:03:26.588498 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578749    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:03:26.588498 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578752    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:03:26.588498 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578755    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:03:26.588498 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578758    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:03:26.588498 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578761    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:03:26.588498 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578763    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:03:26.588498 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578766    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:03:26.588498 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578768    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:03:26.589177 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578771    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:03:26.589177 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578776    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:03:26.589177 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578778    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:03:26.589177 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578781    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:03:26.589177 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578783    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:03:26.589177 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578786    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:03:26.589177 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578790    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:03:26.589177 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578794    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:03:26.589177 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578798    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:03:26.589177 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578801    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:03:26.589177 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578803    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:03:26.589177 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578806    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:03:26.589177 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578809    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:03:26.589177 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578811    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:03:26.589177 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578814    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:03:26.589177 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578816    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:03:26.589177 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578819    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:03:26.589177 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578822    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:03:26.589177 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578824    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:03:26.589730 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578827    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:03:26.589730 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578830    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:03:26.589730 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578833    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:03:26.589730 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578835    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:03:26.589730 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578838    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:03:26.589730 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578840    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:03:26.589730 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578844    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:03:26.589730 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578846    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:03:26.589730 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578849    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:03:26.589730 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578853    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:03:26.589730 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578856    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:03:26.589730 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578860    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:03:26.589730 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578863    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:03:26.589730 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578865    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:03:26.589730 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578869    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:03:26.589730 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578873    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:03:26.589730 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578876    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:03:26.589730 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578879    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:03:26.589730 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578882    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:03:26.590232 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578885    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:03:26.590232 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578888    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:03:26.590232 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578891    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:03:26.590232 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.578894    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:03:26.590232 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.579779    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:03:26.590232 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.586644    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 12:03:26.590232 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.586675    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 12:03:26.590232 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.587264    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 12:03:26.590232 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.587274    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 12:03:26.590232 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587332    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:03:26.590232 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587338    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:03:26.590232 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587343    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:03:26.590232 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587346    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:03:26.590232 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587350    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:03:26.590583 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587353    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:03:26.590583 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587356    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:03:26.590583 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587359    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:03:26.590583 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587361    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:03:26.590583 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587364    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:03:26.590583 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587367    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:03:26.590583 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587370    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:03:26.590583 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587372    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:03:26.590583 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587375    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:03:26.590583 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587378    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:03:26.590583 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587381    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:03:26.590583 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587383    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:03:26.590583 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587386    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:03:26.590583 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587389    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:03:26.590583 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587391    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:03:26.590583 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587394    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:03:26.590583 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587396    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:03:26.590583 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587399    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:03:26.590583 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587401    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:03:26.591070 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587404    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:03:26.591070 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587407    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:03:26.591070 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587409    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:03:26.591070 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587413    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:03:26.591070 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587417    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:03:26.591070 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587421    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:03:26.591070 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587425    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:03:26.591070 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587430    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:03:26.591070 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587433    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:03:26.591070 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587436    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:03:26.591070 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587439    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:03:26.591070 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587441    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:03:26.591070 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587444    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:03:26.591070 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587447    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:03:26.591070 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587450    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:03:26.591070 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587452    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:03:26.591070 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587455    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:03:26.591070 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587458    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:03:26.591070 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587461    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:03:26.591620 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587463    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:03:26.591620 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587467    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:03:26.591620 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587469    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:03:26.591620 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587472    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:03:26.591620 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587475    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:03:26.591620 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587478    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:03:26.591620 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587480    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:03:26.591620 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587483    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:03:26.591620 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587485    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:03:26.591620 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587488    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:03:26.591620 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587491    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:03:26.591620 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587493    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:03:26.591620 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587496    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:03:26.591620 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587498    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:03:26.591620 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587501    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:03:26.591620 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587504    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:03:26.591620 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587507    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:03:26.591620 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587509    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:03:26.591620 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587513    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:03:26.591620 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587516    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:03:26.592127 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587518    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:03:26.592127 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587522    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:03:26.592127 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587524    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:03:26.592127 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587527    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:03:26.592127 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587529    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:03:26.592127 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587532    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:03:26.592127 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587534    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:03:26.592127 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587537    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:03:26.592127 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587539    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:03:26.592127 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587542    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:03:26.592127 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587545    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:03:26.592127 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587547    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:03:26.592127 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587550    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:03:26.592127 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587552    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:03:26.592127 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587555    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:03:26.592127 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587557    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:03:26.592127 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587560    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:03:26.592127 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587562    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:03:26.592127 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587565    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:03:26.592127 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587577    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:03:26.592627 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587580    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:03:26.592627 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587582    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:03:26.592627 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587585    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:03:26.592627 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587588    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:03:26.592627 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587590    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:03:26.592627 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587593    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:03:26.592627 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587596    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:03:26.592627 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587598    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:03:26.592627 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587601    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:03:26.592627 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587604    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:03:26.592627 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587607    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:03:26.592627 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587609    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:03:26.592627 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587613    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:03:26.592627 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587616    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:03:26.592627 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587619    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:03:26.592627 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.587621    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:03:26.593040 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.587626    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:03:26.593040 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588204    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 12:03:26.593040 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588378    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 12:03:26.593040 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588384    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 12:03:26.593040 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588389    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 12:03:26.593040 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588394    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 12:03:26.593040 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588399    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 12:03:26.593040 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588404    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 12:03:26.593040 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588410    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 12:03:26.593040 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588415    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 12:03:26.593040 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588422    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 12:03:26.593040 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588429    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 12:03:26.593040 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588434    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 12:03:26.593040 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588438    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 12:03:26.593040 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588448    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 12:03:26.593418 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588452    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 12:03:26.593418 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588456    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 12:03:26.593418 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588461    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 12:03:26.593418 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588465    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 12:03:26.593418 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588469    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 12:03:26.593418 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588473    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 12:03:26.593418 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588478    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 12:03:26.593418 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588483    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 12:03:26.593418 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588488    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 12:03:26.593418 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588492    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 12:03:26.593418 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588497    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 12:03:26.593418 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588505    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 12:03:26.593418 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588510    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 12:03:26.593418 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588514    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 12:03:26.593418 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588519    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 12:03:26.593418 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588523    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 12:03:26.593418 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588527    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 12:03:26.593418 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588532    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 12:03:26.593418 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588536    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 12:03:26.593901 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588543    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 12:03:26.593901 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588548    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 12:03:26.593901 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588552    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 12:03:26.593901 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588557    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 12:03:26.593901 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588561    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 12:03:26.593901 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588570    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 12:03:26.593901 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588574    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 12:03:26.593901 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588578    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 12:03:26.593901 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588583    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 12:03:26.593901 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588587    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 12:03:26.593901 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588591    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 12:03:26.593901 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588596    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 12:03:26.593901 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588600    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 12:03:26.593901 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588605    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 12:03:26.593901 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588609    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 12:03:26.593901 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588614    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 12:03:26.593901 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588618    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 12:03:26.593901 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588627    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 12:03:26.593901 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588631    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 12:03:26.593901 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588636    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 12:03:26.594401 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588641    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 12:03:26.594401 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588645    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 12:03:26.594401 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588649    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 12:03:26.594401 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588669    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 12:03:26.594401 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588674    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 12:03:26.594401 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588678    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 12:03:26.594401 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588682    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 12:03:26.594401 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588686    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 12:03:26.594401 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588690    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 12:03:26.594401 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588695    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 12:03:26.594401 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588704    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 12:03:26.594401 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588709    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 12:03:26.594401 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588720    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 12:03:26.594401 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588742    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 12:03:26.594401 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588749    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 12:03:26.594401 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588755    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 12:03:26.594401 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588758    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 12:03:26.594401 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588763    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 12:03:26.594401 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588766    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 12:03:26.594401 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588769    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 12:03:26.594965 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588772    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 12:03:26.594965 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588776    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 12:03:26.594965 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588779    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 12:03:26.594965 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588782    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 12:03:26.594965 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588788    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 12:03:26.594965 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588792    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 12:03:26.594965 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588795    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 12:03:26.594965 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588798    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 12:03:26.594965 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588801    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 12:03:26.594965 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588803    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 12:03:26.594965 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588806    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 12:03:26.594965 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588809    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 12:03:26.594965 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588812    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 12:03:26.594965 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588815    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 12:03:26.594965 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588817    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 12:03:26.594965 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588820    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 12:03:26.594965 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588823    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 12:03:26.594965 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588829    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 12:03:26.594965 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588831    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 12:03:26.594965 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588834    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 12:03:26.595472 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588837    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 12:03:26.595472 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588840    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 12:03:26.595472 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588842    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 12:03:26.595472 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588845    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 12:03:26.595472 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588848    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 12:03:26.595472 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:26.588853    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 12:03:26.595472 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.588860    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 12:03:26.595472 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.589212    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 12:03:26.595472 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.591112    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 12:03:26.595472 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.592027    2579 server.go:1019] "Starting client certificate rotation"
Sep 16 12:03:26.595472 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.592120    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:03:26.595472 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.592890    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 12:03:26.595472 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.594403    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:03:26.595940 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.595547    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 12:03:26.606325 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.606304    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 16 12:03:26.609059 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.609042    2579 log.go:25] "Validated CRI v1 image API"
Sep 16 12:03:26.611976 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.611960    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 12:03:26.615024 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.615005    2579 fs.go:135] Filesystem UUIDs: map[2873f9c8-622a-4d50-a6dc-88b3b2978f02:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 eefb4ff0-5638-43ad-99f5-df5c67b1e1a8:/dev/nvme0n1p4]
Sep 16 12:03:26.615103 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.615022    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 12:03:26.621283 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.621163    2579 manager.go:216] Machine: {Timestamp:2026-09-16 12:03:26.619926888 +0000 UTC m=+0.320998206 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100749 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e78a360547ee77b49ff99a5124eb4 SystemUUID:ec2e78a3-6054-7ee7-7b49-ff99a5124eb4 BootID:e023b5ac-ceb6-4774-8a62-609b53479e2c Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:dd:4d:04:43 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:dd:4d:04:43 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:84:d6:f5:f2:37 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 12:03:26.621283 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.621274    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 12:03:26.621411 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.621355    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 12:03:26.621877 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.621856    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:03:26.624442 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.624412    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 12:03:26.624575 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.624446    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-30.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 12:03:26.625110 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.625100    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 12:03:26.625142 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.625114    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 12:03:26.625142 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.625122    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:03:26.626393 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.626382    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 12:03:26.627065 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.627056    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 12:03:26.629651 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.629640    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:03:26.629774 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.629765    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 12:03:26.629809 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.629797    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 12:03:26.629809 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.629808    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 12:03:26.629863 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.629822    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 12:03:26.629863 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.629832    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 16 12:03:26.629863 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.629840    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 12:03:26.630199 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.630188    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:03:26.630253 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.630207    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 12:03:26.634525 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.634503    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 12:03:26.636626 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.636612    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 12:03:26.636706 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.636642    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 12:03:26.638012 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.638001    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 12:03:26.638056 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.638016    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 12:03:26.638056 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.638022    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 12:03:26.638056 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.638028    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 12:03:26.638056 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.638037    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 12:03:26.638056 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.638043    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 12:03:26.638056 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.638057    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 12:03:26.638210 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.638063    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 12:03:26.638210 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.638071    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 12:03:26.638210 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.638076    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 12:03:26.638210 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.638084    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 12:03:26.638210 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.638089    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 12:03:26.638690 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.638681    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 12:03:26.638726 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.638691    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 12:03:26.640239 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:26.640217    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-30.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 12:03:26.640293 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:26.640217    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 12:03:26.641378 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.641360    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-30.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 12:03:26.642135 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.642118    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wll5f"
Sep 16 12:03:26.642743 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.642730    2579 server.go:1268] "Started kubelet"
Sep 16 12:03:26.642806 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.642770    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 12:03:26.642858 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.642813    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 12:03:26.642902 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.642867    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 12:03:26.643552 ip-10-0-137-30 systemd[1]: Started Kubernetes Kubelet.
Sep 16 12:03:26.644057 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.644042    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 12:03:26.646307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.645308    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 12:03:26.647353 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:26.647329    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 12:03:26.648874 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:26.648014    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-30.ec2.internal.18d5cb86c51ced43  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-30.ec2.internal,UID:ip-10-0-137-30.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-30.ec2.internal,},FirstTimestamp:2026-09-16 12:03:26.642703683 +0000 UTC m=+0.343775001,LastTimestamp:2026-09-16 12:03:26.642703683 +0000 UTC m=+0.343775001,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-30.ec2.internal,}"
Sep 16 12:03:26.648950 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.648907    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 12:03:26.649378 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.649360    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 12:03:26.649475 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.649402    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wll5f"
Sep 16 12:03:26.649553 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:26.649535    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-30.ec2.internal\" not found"
Sep 16 12:03:26.650026 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.649970    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 12:03:26.650026 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.649975    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 12:03:26.650026 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.649988    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 12:03:26.650202 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.650064    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 12:03:26.650202 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.650074    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 12:03:26.650419 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.650405    2579 factory.go:55] Registering systemd factory
Sep 16 12:03:26.650455 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.650424    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 16 12:03:26.650629 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.650616    2579 factory.go:153] Registering CRI-O factory
Sep 16 12:03:26.650687 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.650631    2579 factory.go:223] Registration of the crio container factory successfully
Sep 16 12:03:26.650727 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.650693    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 12:03:26.650727 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.650712    2579 factory.go:103] Registering Raw factory
Sep 16 12:03:26.650727 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.650723    2579 manager.go:1196] Started watching for new ooms in manager
Sep 16 12:03:26.651712 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.651697    2579 manager.go:349] Starting recovery of all containers
Sep 16 12:03:26.658518 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.658322    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:03:26.661197 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.661178    2579 manager.go:354] Recovery completed
Sep 16 12:03:26.661405 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:26.661384    2579 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-30.ec2.internal\" not found" node="ip-10-0-137-30.ec2.internal"
Sep 16 12:03:26.667840 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.667828    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:03:26.669944 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.669928    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:03:26.670016 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.669956    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:03:26.670016 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.669967    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-30.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:03:26.670508 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.670492    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 12:03:26.670508 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.670506    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 12:03:26.670635 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.670526    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 12:03:26.670635 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.670497    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:03:26.671304 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.671289    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:03:26.671350 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.671316    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:03:26.671350 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.671328    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-30.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:03:26.671699 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.671683    2579 policy_none.go:49] "None policy: Start"
Sep 16 12:03:26.671764 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.671703    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 12:03:26.671764 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.671716    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 12:03:26.672716 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.672703    2579 policy_none.go:47] "Start"
Sep 16 12:03:26.697934 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.697817    2579 manager.go:337] "Starting Device Plugin manager"
Sep 16 12:03:26.698102 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:26.698085    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 12:03:26.698102 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.698102    2579 server.go:87] "Starting device plugin registration server"
Sep 16 12:03:26.698346 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.698322    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 12:03:26.698346 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.698337    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 12:03:26.699269 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.698494    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 12:03:26.717367 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.699336    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 12:03:26.717367 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.699345    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 12:03:26.717367 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:26.699236    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 12:03:26.717367 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:26.699433    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-30.ec2.internal\" not found"
Sep 16 12:03:26.772474 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.772443    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 12:03:26.773683 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.773664    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 12:03:26.773840 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.773723    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 12:03:26.773840 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.773756    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 12:03:26.773953 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:26.773855    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 12:03:26.775518 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.775501    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:03:26.799426 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.799397    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:03:26.802439 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.802415    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:03:26.802566 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.802454    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:03:26.802566 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.802464    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-30.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:03:26.802566 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.802486    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-30.ec2.internal"
Sep 16 12:03:26.808434 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.808415    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-30.ec2.internal"
Sep 16 12:03:26.808502 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:26.808445    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-30.ec2.internal\": node \"ip-10-0-137-30.ec2.internal\" not found"
Sep 16 12:03:26.822525 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:26.822433    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-30.ec2.internal\" not found"
Sep 16 12:03:26.874707 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.874669    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-30.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-30.ec2.internal"]
Sep 16 12:03:26.874856 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.874752    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:03:26.875798 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.875777    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:03:26.875905 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.875811    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:03:26.875905 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.875828    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-30.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:03:26.877137 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.877124    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:03:26.877273 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.877250    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-30.ec2.internal"
Sep 16 12:03:26.877314 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.877284    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:03:26.877872 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.877856    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:03:26.877956 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.877873    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:03:26.877956 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.877885    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:03:26.877956 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.877894    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:03:26.877956 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.877898    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-30.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:03:26.877956 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.877903    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-30.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:03:26.878980 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.878966    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-30.ec2.internal"
Sep 16 12:03:26.879023 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.878991    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 12:03:26.879711 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.879689    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-30.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 12:03:26.879796 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.879719    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-30.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 12:03:26.879796 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.879732    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-30.ec2.internal" event="NodeHasSufficientPID"
Sep 16 12:03:26.915496 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:26.915471    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-30.ec2.internal\" not found" node="ip-10-0-137-30.ec2.internal"
Sep 16 12:03:26.920050 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:26.920033    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-30.ec2.internal\" not found" node="ip-10-0-137-30.ec2.internal"
Sep 16 12:03:26.923082 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:26.923067    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-30.ec2.internal\" not found"
Sep 16 12:03:26.952519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.952489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e411ae70a2a6889284813c1948077cd9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-30.ec2.internal\" (UID: \"e411ae70a2a6889284813c1948077cd9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-30.ec2.internal"
Sep 16 12:03:26.952519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.952517    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e411ae70a2a6889284813c1948077cd9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-30.ec2.internal\" (UID: \"e411ae70a2a6889284813c1948077cd9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-30.ec2.internal"
Sep 16 12:03:26.952724 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:26.952534    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d94b98261326c611e79bccb709825c5-config\") pod \"kube-apiserver-proxy-ip-10-0-137-30.ec2.internal\" (UID: \"3d94b98261326c611e79bccb709825c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-30.ec2.internal"
Sep 16 12:03:27.023384 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:27.023335    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-30.ec2.internal\" not found"
Sep 16 12:03:27.052682 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:27.052642    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e411ae70a2a6889284813c1948077cd9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-30.ec2.internal\" (UID: \"e411ae70a2a6889284813c1948077cd9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-30.ec2.internal"
Sep 16 12:03:27.052775 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:27.052688    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e411ae70a2a6889284813c1948077cd9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-30.ec2.internal\" (UID: \"e411ae70a2a6889284813c1948077cd9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-30.ec2.internal"
Sep 16 12:03:27.052775 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:27.052726    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d94b98261326c611e79bccb709825c5-config\") pod \"kube-apiserver-proxy-ip-10-0-137-30.ec2.internal\" (UID: \"3d94b98261326c611e79bccb709825c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-30.ec2.internal"
Sep 16 12:03:27.052775 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:27.052732    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e411ae70a2a6889284813c1948077cd9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-30.ec2.internal\" (UID: \"e411ae70a2a6889284813c1948077cd9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-30.ec2.internal"
Sep 16 12:03:27.052775 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:27.052749    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e411ae70a2a6889284813c1948077cd9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-30.ec2.internal\" (UID: \"e411ae70a2a6889284813c1948077cd9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-30.ec2.internal"
Sep 16 12:03:27.052775 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:27.052758    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3d94b98261326c611e79bccb709825c5-config\") pod \"kube-apiserver-proxy-ip-10-0-137-30.ec2.internal\" (UID: \"3d94b98261326c611e79bccb709825c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-30.ec2.internal"
Sep 16 12:03:27.123992 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:27.123945    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-30.ec2.internal\" not found"
Sep 16 12:03:27.220178 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:27.220144    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-30.ec2.internal"
Sep 16 12:03:27.222437 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:27.222411    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-30.ec2.internal"
Sep 16 12:03:27.224383 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:27.224296    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-30.ec2.internal\" not found"
Sep 16 12:03:27.328984 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:27.328951    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-30.ec2.internal\" not found"
Sep 16 12:03:27.434965 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:27.434896    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-30.ec2.internal\" not found"
Sep 16 12:03:27.540717 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:27.540679    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-30.ec2.internal\" not found"
Sep 16 12:03:27.591521 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:27.591485    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 12:03:27.592137 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:27.591657    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 12:03:27.592137 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:27.591656    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 12:03:27.641787 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:27.641751    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-30.ec2.internal\" not found"
Sep 16 12:03:27.653182 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:27.653145    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 11:58:27 +0000 UTC" deadline="2028-07-03 06:43:27.035769936 +0000 UTC"
Sep 16 12:03:27.653182 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:27.653177    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15738h39m59.382596207s"
Sep 16 12:03:27.742844 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:27.742807    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-30.ec2.internal\" not found"
Sep 16 12:03:27.758311 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:27.758284    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:03:27.813874 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:27.813835    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode411ae70a2a6889284813c1948077cd9.slice/crio-02486c979f0a2fa9163f8d18779459416c45b8a8dcf014cea6a61f41b746242b WatchSource:0}: Error finding container 02486c979f0a2fa9163f8d18779459416c45b8a8dcf014cea6a61f41b746242b: Status 404 returned error can't find the container with id 02486c979f0a2fa9163f8d18779459416c45b8a8dcf014cea6a61f41b746242b
Sep 16 12:03:27.820253 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:27.820232    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d94b98261326c611e79bccb709825c5.slice/crio-1f4f193f707398e57b9963c4037fbba0d33ee6265375e4f296aa96fc83231cd4 WatchSource:0}: Error finding container 1f4f193f707398e57b9963c4037fbba0d33ee6265375e4f296aa96fc83231cd4: Status 404 returned error can't find the container with id 1f4f193f707398e57b9963c4037fbba0d33ee6265375e4f296aa96fc83231cd4
Sep 16 12:03:27.820950 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:27.820936    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:03:27.836514 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:27.836491    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:03:27.850413 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:27.850394    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-30.ec2.internal"
Sep 16 12:03:27.864265 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:27.864247    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:03:27.865356 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:27.865343    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-30.ec2.internal"
Sep 16 12:03:27.872181 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:27.872163    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 12:03:28.560206 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.560145    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:03:28.631590 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.631326    2579 apiserver.go:52] "Watching apiserver"
Sep 16 12:03:28.642267 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.642236    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 12:03:28.642729 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.642702    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-hxc2w","openshift-dns/node-resolver-q4hbf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-30.ec2.internal","openshift-multus/network-metrics-daemon-7cn45","openshift-ovn-kubernetes/ovnkube-node-zbbsg","kube-system/global-pull-secret-syncer-qgx6n","openshift-image-registry/node-ca-ldcrr","openshift-multus/multus-additional-cni-plugins-cwn4x","openshift-multus/multus-w64wk","openshift-network-diagnostics/network-check-target-rbck7","openshift-network-operator/iptables-alerter-59h6p","kube-system/konnectivity-agent-htk8n","kube-system/kube-apiserver-proxy-ip-10-0-137-30.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46"]
Sep 16 12:03:28.644377 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.644354    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ldcrr"
Sep 16 12:03:28.645589 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.645568    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-59h6p"
Sep 16 12:03:28.647014 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.646961    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46"
Sep 16 12:03:28.647349 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.647169    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zm2xc\""
Sep 16 12:03:28.647349 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.647223    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 12:03:28.647349 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.647229    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 12:03:28.647564 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.647472    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 12:03:28.648115 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.648093    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:03:28.648115 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.648108    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 12:03:28.648277 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.648200    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tkxmv\""
Sep 16 12:03:28.648536 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.648458    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:28.648536 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.648467    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 12:03:28.648710 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:28.648548    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7cn45" podUID="5441211d-70c6-473a-ac90-417bd7aa57ec"
Sep 16 12:03:28.649447 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.649420    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 12:03:28.649640 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.649616    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 12:03:28.649859 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.649775    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 12:03:28.649859 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.649820    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 12:03:28.650434 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.650200    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7k52n\""
Sep 16 12:03:28.650434 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.650354    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.651943 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.651922    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:28.652039 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:28.651995    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgx6n" podUID="fbbe06db-6359-4c0d-aa65-d07fa8245950"
Sep 16 12:03:28.652958 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.652937    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 12:03:28.653095 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.653083    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 12:03:28.653168 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.653122    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 12:03:28.653230 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.653184    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 12:03:28.653283 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.653246    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 12:03:28.653343 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.653302    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 12:03:28.653343 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.653313    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xfx8j\""
Sep 16 12:03:28.653470 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.653457    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.655103 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.655083    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cwn4x"
Sep 16 12:03:28.656713 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.655960    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 12:03:28.656713 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.656102    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 12:03:28.656713 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.656253    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 12:03:28.656713 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.656370    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 12:03:28.656713 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.656484    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jchx5\""
Sep 16 12:03:28.656713 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.656655    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q4hbf"
Sep 16 12:03:28.657696 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.657543    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 12:03:28.657813 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.657761    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 12:03:28.657953 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.657937    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xvwkp\""
Sep 16 12:03:28.658955 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.658934    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:03:28.659045 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.658978    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 12:03:28.659045 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:28.658996    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbck7" podUID="4fdacb7a-b65c-48f5-b4f3-6580109c3dbd"
Sep 16 12:03:28.659344 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.659325    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 12:03:28.659603 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.659583    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9z6zp\""
Sep 16 12:03:28.660557 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.660532    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-node-log\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.660669 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.660571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-host-cni-bin\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.660669 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.660600    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.660669 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.660627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/977b895b-048d-42e4-854a-fe9d395f0205-device-dir\") pod \"aws-ebs-csi-driver-node-ntn46\" (UID: \"977b895b-048d-42e4-854a-fe9d395f0205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46"
Sep 16 12:03:28.660669 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.660654    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5dhm\" (UniqueName: \"kubernetes.io/projected/5441211d-70c6-473a-ac90-417bd7aa57ec-kube-api-access-k5dhm\") pod \"network-metrics-daemon-7cn45\" (UID: \"5441211d-70c6-473a-ac90-417bd7aa57ec\") " pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:28.660960 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.660679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-systemd-units\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.660960 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.660703    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-host-slash\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.660960 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.660762    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-run-systemd\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.660960 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.660804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-host-cni-netd\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.660960 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.660847    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6cbf941a-3e65-4213-8ab4-fa25f4a27200-ovnkube-script-lib\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.660960 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.660896    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/390dfac1-62d4-4a3f-8586-4bca1a499d37-iptables-alerter-script\") pod \"iptables-alerter-59h6p\" (UID: \"390dfac1-62d4-4a3f-8586-4bca1a499d37\") " pod="openshift-network-operator/iptables-alerter-59h6p"
Sep 16 12:03:28.660960 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.660930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/390dfac1-62d4-4a3f-8586-4bca1a499d37-host-slash\") pod \"iptables-alerter-59h6p\" (UID: \"390dfac1-62d4-4a3f-8586-4bca1a499d37\") " pod="openshift-network-operator/iptables-alerter-59h6p"
Sep 16 12:03:28.660960 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.660958    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6a91efe-a2f3-4829-9777-1ea089f08030-host\") pod \"node-ca-ldcrr\" (UID: \"a6a91efe-a2f3-4829-9777-1ea089f08030\") " pod="openshift-image-registry/node-ca-ldcrr"
Sep 16 12:03:28.661392 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.660973    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.661392 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.660983    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-run-openvswitch\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.661392 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.661008    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-log-socket\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.661392 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.661039    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/977b895b-048d-42e4-854a-fe9d395f0205-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ntn46\" (UID: \"977b895b-048d-42e4-854a-fe9d395f0205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46"
Sep 16 12:03:28.661392 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.661080    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/977b895b-048d-42e4-854a-fe9d395f0205-socket-dir\") pod \"aws-ebs-csi-driver-node-ntn46\" (UID: \"977b895b-048d-42e4-854a-fe9d395f0205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46"
Sep 16 12:03:28.661392 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.661132    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/977b895b-048d-42e4-854a-fe9d395f0205-registration-dir\") pod \"aws-ebs-csi-driver-node-ntn46\" (UID: \"977b895b-048d-42e4-854a-fe9d395f0205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46"
Sep 16 12:03:28.661392 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.661159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-host-kubelet\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.661392 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.661190    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-var-lib-openvswitch\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.661392 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.661216    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-etc-openvswitch\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.661392 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.661239    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a6a91efe-a2f3-4829-9777-1ea089f08030-serviceca\") pod \"node-ca-ldcrr\" (UID: \"a6a91efe-a2f3-4829-9777-1ea089f08030\") " pod="openshift-image-registry/node-ca-ldcrr"
Sep 16 12:03:28.661392 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.661264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/977b895b-048d-42e4-854a-fe9d395f0205-sys-fs\") pod \"aws-ebs-csi-driver-node-ntn46\" (UID: \"977b895b-048d-42e4-854a-fe9d395f0205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46"
Sep 16 12:03:28.661392 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.661295    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6cbf941a-3e65-4213-8ab4-fa25f4a27200-env-overrides\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.661392 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.661311    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75wt4\" (UniqueName: \"kubernetes.io/projected/390dfac1-62d4-4a3f-8586-4bca1a499d37-kube-api-access-75wt4\") pod \"iptables-alerter-59h6p\" (UID: \"390dfac1-62d4-4a3f-8586-4bca1a499d37\") " pod="openshift-network-operator/iptables-alerter-59h6p"
Sep 16 12:03:28.661392 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.661329    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs\") pod \"network-metrics-daemon-7cn45\" (UID: \"5441211d-70c6-473a-ac90-417bd7aa57ec\") " pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:28.661392 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.661345    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-host-run-netns\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.661392 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.661360    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6cbf941a-3e65-4213-8ab4-fa25f4a27200-ovnkube-config\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.661392 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.661376    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6cbf941a-3e65-4213-8ab4-fa25f4a27200-ovn-node-metrics-cert\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.662140 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.661437    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb7dv\" (UniqueName: \"kubernetes.io/projected/6cbf941a-3e65-4213-8ab4-fa25f4a27200-kube-api-access-cb7dv\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.662140 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.661472    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-run-ovn\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.662140 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.661510    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-host-run-ovn-kubernetes\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.662140 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.661544    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj6z6\" (UniqueName: \"kubernetes.io/projected/a6a91efe-a2f3-4829-9777-1ea089f08030-kube-api-access-fj6z6\") pod \"node-ca-ldcrr\" (UID: \"a6a91efe-a2f3-4829-9777-1ea089f08030\") " pod="openshift-image-registry/node-ca-ldcrr"
Sep 16 12:03:28.662140 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.661578    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/977b895b-048d-42e4-854a-fe9d395f0205-etc-selinux\") pod \"aws-ebs-csi-driver-node-ntn46\" (UID: \"977b895b-048d-42e4-854a-fe9d395f0205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46"
Sep 16 12:03:28.662140 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.661614    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhwfw\" (UniqueName: \"kubernetes.io/projected/977b895b-048d-42e4-854a-fe9d395f0205-kube-api-access-xhwfw\") pod \"aws-ebs-csi-driver-node-ntn46\" (UID: \"977b895b-048d-42e4-854a-fe9d395f0205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46"
Sep 16 12:03:28.662488 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.662225    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-htk8n"
Sep 16 12:03:28.663543 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.663522    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:03:28.663639 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.663555    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fmwkh\""
Sep 16 12:03:28.663865 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.663845    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 12:03:28.665953 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.665934    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 12:03:28.666270 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.666247    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7xhgt\""
Sep 16 12:03:28.666556 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.666537    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 12:03:28.666632 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.666599    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 12:03:28.701123 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.701096    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kmk9t"
Sep 16 12:03:28.710739 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.710710    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kmk9t"
Sep 16 12:03:28.751608 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.751577    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 12:03:28.762049 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs\") pod \"network-metrics-daemon-7cn45\" (UID: \"5441211d-70c6-473a-ac90-417bd7aa57ec\") " pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:28.762049 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-host-run-netns\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.762290 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6cbf941a-3e65-4213-8ab4-fa25f4a27200-ovnkube-config\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.762290 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762114    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb7dv\" (UniqueName: \"kubernetes.io/projected/6cbf941a-3e65-4213-8ab4-fa25f4a27200-kube-api-access-cb7dv\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.762290 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2e0fe6c2-78d6-4a99-8197-50e62073f615-etc-tuned\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.762290 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762166    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e0fe6c2-78d6-4a99-8197-50e62073f615-tmp\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.762290 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsbrp\" (UniqueName: \"kubernetes.io/projected/2e0fe6c2-78d6-4a99-8197-50e62073f615-kube-api-access-zsbrp\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.762290 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762213    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-host-run-netns\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.762290 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762220    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-run-ovn\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.762537 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762303    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/977b895b-048d-42e4-854a-fe9d395f0205-etc-selinux\") pod \"aws-ebs-csi-driver-node-ntn46\" (UID: \"977b895b-048d-42e4-854a-fe9d395f0205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46"
Sep 16 12:03:28.762537 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9ght\" (UniqueName: \"kubernetes.io/projected/fc93b327-87ec-4a56-a549-9dbfddce7fc0-kube-api-access-k9ght\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.762537 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762364    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-etc-modprobe-d\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.762537 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762386    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-run-ovn\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.762537 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762390    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-sys\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.762537 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-host-var-lib-kubelet\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.762537 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762452    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/977b895b-048d-42e4-854a-fe9d395f0205-etc-selinux\") pod \"aws-ebs-csi-driver-node-ntn46\" (UID: \"977b895b-048d-42e4-854a-fe9d395f0205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46"
Sep 16 12:03:28.762537 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762467    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-host-cni-bin\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.762537 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:28.762474    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:28.762537 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/977b895b-048d-42e4-854a-fe9d395f0205-device-dir\") pod \"aws-ebs-csi-driver-node-ntn46\" (UID: \"977b895b-048d-42e4-854a-fe9d395f0205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46"
Sep 16 12:03:28.762537 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762512    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-host-cni-bin\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.762537 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762520    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-etc-sysconfig\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.762951 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:28.762552    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs podName:5441211d-70c6-473a-ac90-417bd7aa57ec nodeName:}" failed. No retries permitted until 2026-09-16 12:03:29.26253044 +0000 UTC m=+2.963601770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs") pod "network-metrics-daemon-7cn45" (UID: "5441211d-70c6-473a-ac90-417bd7aa57ec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:28.762951 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762570    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-etc-systemd\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.762951 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762596    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpfv9\" (UniqueName: \"kubernetes.io/projected/4fdacb7a-b65c-48f5-b4f3-6580109c3dbd-kube-api-access-bpfv9\") pod \"network-check-target-rbck7\" (UID: \"4fdacb7a-b65c-48f5-b4f3-6580109c3dbd\") " pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:03:28.762951 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762622    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5dhm\" (UniqueName: \"kubernetes.io/projected/5441211d-70c6-473a-ac90-417bd7aa57ec-kube-api-access-k5dhm\") pod \"network-metrics-daemon-7cn45\" (UID: \"5441211d-70c6-473a-ac90-417bd7aa57ec\") " pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:28.762951 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762649    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-run-systemd\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.762951 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762651    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/977b895b-048d-42e4-854a-fe9d395f0205-device-dir\") pod \"aws-ebs-csi-driver-node-ntn46\" (UID: \"977b895b-048d-42e4-854a-fe9d395f0205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46"
Sep 16 12:03:28.762951 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762674    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-host-cni-netd\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.762951 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/390dfac1-62d4-4a3f-8586-4bca1a499d37-iptables-alerter-script\") pod \"iptables-alerter-59h6p\" (UID: \"390dfac1-62d4-4a3f-8586-4bca1a499d37\") " pod="openshift-network-operator/iptables-alerter-59h6p"
Sep 16 12:03:28.762951 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/390dfac1-62d4-4a3f-8586-4bca1a499d37-host-slash\") pod \"iptables-alerter-59h6p\" (UID: \"390dfac1-62d4-4a3f-8586-4bca1a499d37\") " pod="openshift-network-operator/iptables-alerter-59h6p"
Sep 16 12:03:28.762951 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762777    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-host-cni-netd\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.762951 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762776    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fbbe06db-6359-4c0d-aa65-d07fa8245950-kubelet-config\") pod \"global-pull-secret-syncer-qgx6n\" (UID: \"fbbe06db-6359-4c0d-aa65-d07fa8245950\") " pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:28.762951 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762806    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-run-openvswitch\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.762951 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-log-socket\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.762951 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762845    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/977b895b-048d-42e4-854a-fe9d395f0205-registration-dir\") pod \"aws-ebs-csi-driver-node-ntn46\" (UID: \"977b895b-048d-42e4-854a-fe9d395f0205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46"
Sep 16 12:03:28.762951 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762862    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-etc-kubernetes\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.762951 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762859    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6cbf941a-3e65-4213-8ab4-fa25f4a27200-ovnkube-config\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.763686 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762900    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-log-socket\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.763686 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762922    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-run-openvswitch\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.763686 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762753    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-run-systemd\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.763686 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762938    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-run\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.763686 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762954    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/390dfac1-62d4-4a3f-8586-4bca1a499d37-host-slash\") pod \"iptables-alerter-59h6p\" (UID: \"390dfac1-62d4-4a3f-8586-4bca1a499d37\") " pod="openshift-network-operator/iptables-alerter-59h6p"
Sep 16 12:03:28.763686 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762965    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-host-kubelet\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.763686 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.762988    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-var-lib-openvswitch\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.763686 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763010    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-etc-openvswitch\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.763686 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a6a91efe-a2f3-4829-9777-1ea089f08030-serviceca\") pod \"node-ca-ldcrr\" (UID: \"a6a91efe-a2f3-4829-9777-1ea089f08030\") " pod="openshift-image-registry/node-ca-ldcrr"
Sep 16 12:03:28.763686 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/977b895b-048d-42e4-854a-fe9d395f0205-sys-fs\") pod \"aws-ebs-csi-driver-node-ntn46\" (UID: \"977b895b-048d-42e4-854a-fe9d395f0205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46"
Sep 16 12:03:28.763686 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763062    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/977b895b-048d-42e4-854a-fe9d395f0205-registration-dir\") pod \"aws-ebs-csi-driver-node-ntn46\" (UID: \"977b895b-048d-42e4-854a-fe9d395f0205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46"
Sep 16 12:03:28.763686 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763075    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-etc-openvswitch\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.763686 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763082    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-system-cni-dir\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.763686 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763079    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-var-lib-openvswitch\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.763686 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763091    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-host-kubelet\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.763686 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-hostroot\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.763686 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/977b895b-048d-42e4-854a-fe9d395f0205-sys-fs\") pod \"aws-ebs-csi-driver-node-ntn46\" (UID: \"977b895b-048d-42e4-854a-fe9d395f0205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46"
Sep 16 12:03:28.764558 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fc93b327-87ec-4a56-a549-9dbfddce7fc0-multus-daemon-config\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.764558 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763175    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-cnibin\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.764558 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-host-var-lib-cni-bin\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.764558 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-host-run-multus-certs\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.764558 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763251    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6180fb10-c394-4bca-97af-1a28211ea33f-cnibin\") pod \"multus-additional-cni-plugins-cwn4x\" (UID: \"6180fb10-c394-4bca-97af-1a28211ea33f\") " pod="openshift-multus/multus-additional-cni-plugins-cwn4x"
Sep 16 12:03:28.764558 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6180fb10-c394-4bca-97af-1a28211ea33f-cni-binary-copy\") pod \"multus-additional-cni-plugins-cwn4x\" (UID: \"6180fb10-c394-4bca-97af-1a28211ea33f\") " pod="openshift-multus/multus-additional-cni-plugins-cwn4x"
Sep 16 12:03:28.764558 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763347    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/390dfac1-62d4-4a3f-8586-4bca1a499d37-iptables-alerter-script\") pod \"iptables-alerter-59h6p\" (UID: \"390dfac1-62d4-4a3f-8586-4bca1a499d37\") " pod="openshift-network-operator/iptables-alerter-59h6p"
Sep 16 12:03:28.764558 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763347    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-etc-sysctl-d\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.764558 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763405    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-host\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.764558 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjxp5\" (UniqueName: \"kubernetes.io/projected/b38a0789-177d-497c-8d7f-7d7177f5015e-kube-api-access-wjxp5\") pod \"node-resolver-q4hbf\" (UID: \"b38a0789-177d-497c-8d7f-7d7177f5015e\") " pod="openshift-dns/node-resolver-q4hbf"
Sep 16 12:03:28.764558 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763472    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75wt4\" (UniqueName: \"kubernetes.io/projected/390dfac1-62d4-4a3f-8586-4bca1a499d37-kube-api-access-75wt4\") pod \"iptables-alerter-59h6p\" (UID: \"390dfac1-62d4-4a3f-8586-4bca1a499d37\") " pod="openshift-network-operator/iptables-alerter-59h6p"
Sep 16 12:03:28.764558 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763497    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6180fb10-c394-4bca-97af-1a28211ea33f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cwn4x\" (UID: \"6180fb10-c394-4bca-97af-1a28211ea33f\") " pod="openshift-multus/multus-additional-cni-plugins-cwn4x"
Sep 16 12:03:28.764558 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763519    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6cbf941a-3e65-4213-8ab4-fa25f4a27200-ovn-node-metrics-cert\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.764558 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763556    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/14653c0f-3cc1-4527-b610-bb5c65e4f7db-konnectivity-ca\") pod \"konnectivity-agent-htk8n\" (UID: \"14653c0f-3cc1-4527-b610-bb5c65e4f7db\") " pod="kube-system/konnectivity-agent-htk8n"
Sep 16 12:03:28.764558 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763588    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-multus-socket-dir-parent\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.764558 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763612    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-host-run-netns\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.765380 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-host-run-ovn-kubernetes\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.765380 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763682    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj6z6\" (UniqueName: \"kubernetes.io/projected/a6a91efe-a2f3-4829-9777-1ea089f08030-kube-api-access-fj6z6\") pod \"node-ca-ldcrr\" (UID: \"a6a91efe-a2f3-4829-9777-1ea089f08030\") " pod="openshift-image-registry/node-ca-ldcrr"
Sep 16 12:03:28.765380 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763706    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-host-run-ovn-kubernetes\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.765380 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763714    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhwfw\" (UniqueName: \"kubernetes.io/projected/977b895b-048d-42e4-854a-fe9d395f0205-kube-api-access-xhwfw\") pod \"aws-ebs-csi-driver-node-ntn46\" (UID: \"977b895b-048d-42e4-854a-fe9d395f0205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46"
Sep 16 12:03:28.765380 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763762    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-lib-modules\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.765380 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fbbe06db-6359-4c0d-aa65-d07fa8245950-dbus\") pod \"global-pull-secret-syncer-qgx6n\" (UID: \"fbbe06db-6359-4c0d-aa65-d07fa8245950\") " pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:28.765380 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-node-log\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.765380 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763850    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.765380 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763880    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc93b327-87ec-4a56-a549-9dbfddce7fc0-cni-binary-copy\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.765380 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763908    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-host-run-k8s-cni-cncf-io\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.765380 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763940    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-node-log\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.765380 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-systemd-units\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.765380 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764003    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.765380 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.763998    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-host-slash\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.765380 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764038    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6cbf941a-3e65-4213-8ab4-fa25f4a27200-ovnkube-script-lib\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.765380 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764052    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 12:03:28.765380 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/977b895b-048d-42e4-854a-fe9d395f0205-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ntn46\" (UID: \"977b895b-048d-42e4-854a-fe9d395f0205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46"
Sep 16 12:03:28.766238 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764077    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-host-slash\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.766238 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/977b895b-048d-42e4-854a-fe9d395f0205-socket-dir\") pod \"aws-ebs-csi-driver-node-ntn46\" (UID: \"977b895b-048d-42e4-854a-fe9d395f0205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46"
Sep 16 12:03:28.766238 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764111    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6cbf941a-3e65-4213-8ab4-fa25f4a27200-systemd-units\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.766238 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764140    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-multus-conf-dir\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.766238 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764169    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6180fb10-c394-4bca-97af-1a28211ea33f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cwn4x\" (UID: \"6180fb10-c394-4bca-97af-1a28211ea33f\") " pod="openshift-multus/multus-additional-cni-plugins-cwn4x"
Sep 16 12:03:28.766238 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b38a0789-177d-497c-8d7f-7d7177f5015e-hosts-file\") pod \"node-resolver-q4hbf\" (UID: \"b38a0789-177d-497c-8d7f-7d7177f5015e\") " pod="openshift-dns/node-resolver-q4hbf"
Sep 16 12:03:28.766238 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764216    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/977b895b-048d-42e4-854a-fe9d395f0205-socket-dir\") pod \"aws-ebs-csi-driver-node-ntn46\" (UID: \"977b895b-048d-42e4-854a-fe9d395f0205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46"
Sep 16 12:03:28.766238 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6a91efe-a2f3-4829-9777-1ea089f08030-host\") pod \"node-ca-ldcrr\" (UID: \"a6a91efe-a2f3-4829-9777-1ea089f08030\") " pod="openshift-image-registry/node-ca-ldcrr"
Sep 16 12:03:28.766238 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764252    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmgkf\" (UniqueName: \"kubernetes.io/projected/6180fb10-c394-4bca-97af-1a28211ea33f-kube-api-access-wmgkf\") pod \"multus-additional-cni-plugins-cwn4x\" (UID: \"6180fb10-c394-4bca-97af-1a28211ea33f\") " pod="openshift-multus/multus-additional-cni-plugins-cwn4x"
Sep 16 12:03:28.766238 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764275    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/977b895b-048d-42e4-854a-fe9d395f0205-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ntn46\" (UID: \"977b895b-048d-42e4-854a-fe9d395f0205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46"
Sep 16 12:03:28.766238 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764280    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-etc-sysctl-conf\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.766238 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764311    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/14653c0f-3cc1-4527-b610-bb5c65e4f7db-agent-certs\") pod \"konnectivity-agent-htk8n\" (UID: \"14653c0f-3cc1-4527-b610-bb5c65e4f7db\") " pod="kube-system/konnectivity-agent-htk8n"
Sep 16 12:03:28.766238 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764337    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6180fb10-c394-4bca-97af-1a28211ea33f-system-cni-dir\") pod \"multus-additional-cni-plugins-cwn4x\" (UID: \"6180fb10-c394-4bca-97af-1a28211ea33f\") " pod="openshift-multus/multus-additional-cni-plugins-cwn4x"
Sep 16 12:03:28.766238 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764365    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6180fb10-c394-4bca-97af-1a28211ea33f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cwn4x\" (UID: \"6180fb10-c394-4bca-97af-1a28211ea33f\") " pod="openshift-multus/multus-additional-cni-plugins-cwn4x"
Sep 16 12:03:28.766238 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764356    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6a91efe-a2f3-4829-9777-1ea089f08030-host\") pod \"node-ca-ldcrr\" (UID: \"a6a91efe-a2f3-4829-9777-1ea089f08030\") " pod="openshift-image-registry/node-ca-ldcrr"
Sep 16 12:03:28.766238 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764394    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-var-lib-kubelet\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.766238 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764419    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b38a0789-177d-497c-8d7f-7d7177f5015e-tmp-dir\") pod \"node-resolver-q4hbf\" (UID: \"b38a0789-177d-497c-8d7f-7d7177f5015e\") " pod="openshift-dns/node-resolver-q4hbf"
Sep 16 12:03:28.767110 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6cbf941a-3e65-4213-8ab4-fa25f4a27200-env-overrides\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.767110 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764474    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-multus-cni-dir\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.767110 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-host-var-lib-cni-multus\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.767110 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-etc-kubernetes\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.767110 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fbbe06db-6359-4c0d-aa65-d07fa8245950-original-pull-secret\") pod \"global-pull-secret-syncer-qgx6n\" (UID: \"fbbe06db-6359-4c0d-aa65-d07fa8245950\") " pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:28.767110 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764599    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6cbf941a-3e65-4213-8ab4-fa25f4a27200-ovnkube-script-lib\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.767110 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.764551    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a6a91efe-a2f3-4829-9777-1ea089f08030-serviceca\") pod \"node-ca-ldcrr\" (UID: \"a6a91efe-a2f3-4829-9777-1ea089f08030\") " pod="openshift-image-registry/node-ca-ldcrr"
Sep 16 12:03:28.767110 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.765018    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6cbf941a-3e65-4213-8ab4-fa25f4a27200-env-overrides\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.769247 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.769218    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6cbf941a-3e65-4213-8ab4-fa25f4a27200-ovn-node-metrics-cert\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.778189 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.778146    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5dhm\" (UniqueName: \"kubernetes.io/projected/5441211d-70c6-473a-ac90-417bd7aa57ec-kube-api-access-k5dhm\") pod \"network-metrics-daemon-7cn45\" (UID: \"5441211d-70c6-473a-ac90-417bd7aa57ec\") " pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:28.778865 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.778807    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb7dv\" (UniqueName: \"kubernetes.io/projected/6cbf941a-3e65-4213-8ab4-fa25f4a27200-kube-api-access-cb7dv\") pod \"ovnkube-node-zbbsg\" (UID: \"6cbf941a-3e65-4213-8ab4-fa25f4a27200\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.779163 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.779109    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhwfw\" (UniqueName: \"kubernetes.io/projected/977b895b-048d-42e4-854a-fe9d395f0205-kube-api-access-xhwfw\") pod \"aws-ebs-csi-driver-node-ntn46\" (UID: \"977b895b-048d-42e4-854a-fe9d395f0205\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46"
Sep 16 12:03:28.779372 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.779324    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-30.ec2.internal" event={"ID":"e411ae70a2a6889284813c1948077cd9","Type":"ContainerStarted","Data":"02486c979f0a2fa9163f8d18779459416c45b8a8dcf014cea6a61f41b746242b"}
Sep 16 12:03:28.779442 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.779355    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75wt4\" (UniqueName: \"kubernetes.io/projected/390dfac1-62d4-4a3f-8586-4bca1a499d37-kube-api-access-75wt4\") pod \"iptables-alerter-59h6p\" (UID: \"390dfac1-62d4-4a3f-8586-4bca1a499d37\") " pod="openshift-network-operator/iptables-alerter-59h6p"
Sep 16 12:03:28.779995 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.779961    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj6z6\" (UniqueName: \"kubernetes.io/projected/a6a91efe-a2f3-4829-9777-1ea089f08030-kube-api-access-fj6z6\") pod \"node-ca-ldcrr\" (UID: \"a6a91efe-a2f3-4829-9777-1ea089f08030\") " pod="openshift-image-registry/node-ca-ldcrr"
Sep 16 12:03:28.780652 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.780631    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-30.ec2.internal" event={"ID":"3d94b98261326c611e79bccb709825c5","Type":"ContainerStarted","Data":"1f4f193f707398e57b9963c4037fbba0d33ee6265375e4f296aa96fc83231cd4"}
Sep 16 12:03:28.865379 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/14653c0f-3cc1-4527-b610-bb5c65e4f7db-agent-certs\") pod \"konnectivity-agent-htk8n\" (UID: \"14653c0f-3cc1-4527-b610-bb5c65e4f7db\") " pod="kube-system/konnectivity-agent-htk8n"
Sep 16 12:03:28.865379 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865382    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6180fb10-c394-4bca-97af-1a28211ea33f-system-cni-dir\") pod \"multus-additional-cni-plugins-cwn4x\" (UID: \"6180fb10-c394-4bca-97af-1a28211ea33f\") " pod="openshift-multus/multus-additional-cni-plugins-cwn4x"
Sep 16 12:03:28.865651 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6180fb10-c394-4bca-97af-1a28211ea33f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cwn4x\" (UID: \"6180fb10-c394-4bca-97af-1a28211ea33f\") " pod="openshift-multus/multus-additional-cni-plugins-cwn4x"
Sep 16 12:03:28.865651 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865424    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-var-lib-kubelet\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.865651 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865451    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b38a0789-177d-497c-8d7f-7d7177f5015e-tmp-dir\") pod \"node-resolver-q4hbf\" (UID: \"b38a0789-177d-497c-8d7f-7d7177f5015e\") " pod="openshift-dns/node-resolver-q4hbf"
Sep 16 12:03:28.865651 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865478    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-multus-cni-dir\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.865651 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-host-var-lib-cni-multus\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.865651 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865497    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6180fb10-c394-4bca-97af-1a28211ea33f-system-cni-dir\") pod \"multus-additional-cni-plugins-cwn4x\" (UID: \"6180fb10-c394-4bca-97af-1a28211ea33f\") " pod="openshift-multus/multus-additional-cni-plugins-cwn4x"
Sep 16 12:03:28.865651 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865515    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-etc-kubernetes\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.865651 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fbbe06db-6359-4c0d-aa65-d07fa8245950-original-pull-secret\") pod \"global-pull-secret-syncer-qgx6n\" (UID: \"fbbe06db-6359-4c0d-aa65-d07fa8245950\") " pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:28.865651 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865563    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2e0fe6c2-78d6-4a99-8197-50e62073f615-etc-tuned\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.865651 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865578    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e0fe6c2-78d6-4a99-8197-50e62073f615-tmp\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.865651 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865593    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsbrp\" (UniqueName: \"kubernetes.io/projected/2e0fe6c2-78d6-4a99-8197-50e62073f615-kube-api-access-zsbrp\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.865651 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865613    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9ght\" (UniqueName: \"kubernetes.io/projected/fc93b327-87ec-4a56-a549-9dbfddce7fc0-kube-api-access-k9ght\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.865651 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865628    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-etc-modprobe-d\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.865651 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865643    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-sys\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.866379 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-host-var-lib-kubelet\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.866379 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-etc-sysconfig\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.866379 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865744    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-etc-systemd\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.866379 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpfv9\" (UniqueName: \"kubernetes.io/projected/4fdacb7a-b65c-48f5-b4f3-6580109c3dbd-kube-api-access-bpfv9\") pod \"network-check-target-rbck7\" (UID: \"4fdacb7a-b65c-48f5-b4f3-6580109c3dbd\") " pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:03:28.866379 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865785    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fbbe06db-6359-4c0d-aa65-d07fa8245950-kubelet-config\") pod \"global-pull-secret-syncer-qgx6n\" (UID: \"fbbe06db-6359-4c0d-aa65-d07fa8245950\") " pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:28.866379 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865803    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-etc-kubernetes\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.866379 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-run\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.866379 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-system-cni-dir\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.866379 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-hostroot\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.866379 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865868    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fc93b327-87ec-4a56-a549-9dbfddce7fc0-multus-daemon-config\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.866379 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865887    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-cnibin\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.866379 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b38a0789-177d-497c-8d7f-7d7177f5015e-tmp-dir\") pod \"node-resolver-q4hbf\" (UID: \"b38a0789-177d-497c-8d7f-7d7177f5015e\") " pod="openshift-dns/node-resolver-q4hbf"
Sep 16 12:03:28.866379 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865911    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-host-var-lib-cni-bin\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.866379 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865944    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-host-run-multus-certs\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.866379 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865960    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-sys\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.866379 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865969    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6180fb10-c394-4bca-97af-1a28211ea33f-cnibin\") pod \"multus-additional-cni-plugins-cwn4x\" (UID: \"6180fb10-c394-4bca-97af-1a28211ea33f\") " pod="openshift-multus/multus-additional-cni-plugins-cwn4x"
Sep 16 12:03:28.866379 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866007    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6180fb10-c394-4bca-97af-1a28211ea33f-cnibin\") pod \"multus-additional-cni-plugins-cwn4x\" (UID: \"6180fb10-c394-4bca-97af-1a28211ea33f\") " pod="openshift-multus/multus-additional-cni-plugins-cwn4x"
Sep 16 12:03:28.866379 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866021    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6180fb10-c394-4bca-97af-1a28211ea33f-cni-binary-copy\") pod \"multus-additional-cni-plugins-cwn4x\" (UID: \"6180fb10-c394-4bca-97af-1a28211ea33f\") " pod="openshift-multus/multus-additional-cni-plugins-cwn4x"
Sep 16 12:03:28.867352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866035    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-run\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.867352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-etc-sysctl-d\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.867352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866080    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-etc-systemd\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.867352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6180fb10-c394-4bca-97af-1a28211ea33f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cwn4x\" (UID: \"6180fb10-c394-4bca-97af-1a28211ea33f\") " pod="openshift-multus/multus-additional-cni-plugins-cwn4x"
Sep 16 12:03:28.867352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866143    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-etc-sysctl-d\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.867352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866144    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-host-var-lib-kubelet\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.867352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866128    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-system-cni-dir\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.867352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866197    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-etc-sysconfig\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.867352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865561    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-multus-cni-dir\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.867352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866237    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fbbe06db-6359-4c0d-aa65-d07fa8245950-kubelet-config\") pod \"global-pull-secret-syncer-qgx6n\" (UID: \"fbbe06db-6359-4c0d-aa65-d07fa8245950\") " pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:28.867352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866348    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-etc-kubernetes\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.867352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866384    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-host\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.867352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866391    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-hostroot\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.867352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866447    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-host-var-lib-cni-bin\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.867352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.865506    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-var-lib-kubelet\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.867352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866197    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-host-run-multus-certs\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.867352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866502    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-cnibin\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.867352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866525    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-host-var-lib-cni-multus\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.868289 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866545    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjxp5\" (UniqueName: \"kubernetes.io/projected/b38a0789-177d-497c-8d7f-7d7177f5015e-kube-api-access-wjxp5\") pod \"node-resolver-q4hbf\" (UID: \"b38a0789-177d-497c-8d7f-7d7177f5015e\") " pod="openshift-dns/node-resolver-q4hbf"
Sep 16 12:03:28.868289 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866589    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-etc-kubernetes\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.868289 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866600    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-etc-modprobe-d\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.868289 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866630    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-host\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.868289 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6180fb10-c394-4bca-97af-1a28211ea33f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cwn4x\" (UID: \"6180fb10-c394-4bca-97af-1a28211ea33f\") " pod="openshift-multus/multus-additional-cni-plugins-cwn4x"
Sep 16 12:03:28.868289 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:28.866701    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:28.868289 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866743    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fc93b327-87ec-4a56-a549-9dbfddce7fc0-multus-daemon-config\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.868289 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/14653c0f-3cc1-4527-b610-bb5c65e4f7db-konnectivity-ca\") pod \"konnectivity-agent-htk8n\" (UID: \"14653c0f-3cc1-4527-b610-bb5c65e4f7db\") " pod="kube-system/konnectivity-agent-htk8n"
Sep 16 12:03:28.868289 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-multus-socket-dir-parent\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.868289 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:28.866840    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbbe06db-6359-4c0d-aa65-d07fa8245950-original-pull-secret podName:fbbe06db-6359-4c0d-aa65-d07fa8245950 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:29.366818825 +0000 UTC m=+3.067890143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fbbe06db-6359-4c0d-aa65-d07fa8245950-original-pull-secret") pod "global-pull-secret-syncer-qgx6n" (UID: "fbbe06db-6359-4c0d-aa65-d07fa8245950") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:28.868289 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866857    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-multus-socket-dir-parent\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.868289 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866889    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-host-run-netns\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.868289 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866898    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6180fb10-c394-4bca-97af-1a28211ea33f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cwn4x\" (UID: \"6180fb10-c394-4bca-97af-1a28211ea33f\") " pod="openshift-multus/multus-additional-cni-plugins-cwn4x"
Sep 16 12:03:28.868289 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866900    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6180fb10-c394-4bca-97af-1a28211ea33f-cni-binary-copy\") pod \"multus-additional-cni-plugins-cwn4x\" (UID: \"6180fb10-c394-4bca-97af-1a28211ea33f\") " pod="openshift-multus/multus-additional-cni-plugins-cwn4x"
Sep 16 12:03:28.868289 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-lib-modules\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.868289 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866943    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-host-run-netns\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.868289 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866951    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fbbe06db-6359-4c0d-aa65-d07fa8245950-dbus\") pod \"global-pull-secret-syncer-qgx6n\" (UID: \"fbbe06db-6359-4c0d-aa65-d07fa8245950\") " pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:28.869070 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.866979    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc93b327-87ec-4a56-a549-9dbfddce7fc0-cni-binary-copy\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.869070 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.867004    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-host-run-k8s-cni-cncf-io\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.869070 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.867023    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-lib-modules\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.869070 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.867036    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-multus-conf-dir\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.869070 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.867041    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fbbe06db-6359-4c0d-aa65-d07fa8245950-dbus\") pod \"global-pull-secret-syncer-qgx6n\" (UID: \"fbbe06db-6359-4c0d-aa65-d07fa8245950\") " pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:28.869070 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.867065    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6180fb10-c394-4bca-97af-1a28211ea33f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cwn4x\" (UID: \"6180fb10-c394-4bca-97af-1a28211ea33f\") " pod="openshift-multus/multus-additional-cni-plugins-cwn4x"
Sep 16 12:03:28.869070 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.867081    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-host-run-k8s-cni-cncf-io\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.869070 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.867100    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc93b327-87ec-4a56-a549-9dbfddce7fc0-multus-conf-dir\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.869070 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.867093    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b38a0789-177d-497c-8d7f-7d7177f5015e-hosts-file\") pod \"node-resolver-q4hbf\" (UID: \"b38a0789-177d-497c-8d7f-7d7177f5015e\") " pod="openshift-dns/node-resolver-q4hbf"
Sep 16 12:03:28.869070 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.867147    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b38a0789-177d-497c-8d7f-7d7177f5015e-hosts-file\") pod \"node-resolver-q4hbf\" (UID: \"b38a0789-177d-497c-8d7f-7d7177f5015e\") " pod="openshift-dns/node-resolver-q4hbf"
Sep 16 12:03:28.869070 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.867174    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmgkf\" (UniqueName: \"kubernetes.io/projected/6180fb10-c394-4bca-97af-1a28211ea33f-kube-api-access-wmgkf\") pod \"multus-additional-cni-plugins-cwn4x\" (UID: \"6180fb10-c394-4bca-97af-1a28211ea33f\") " pod="openshift-multus/multus-additional-cni-plugins-cwn4x"
Sep 16 12:03:28.869070 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.867211    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-etc-sysctl-conf\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.869070 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.867336    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/14653c0f-3cc1-4527-b610-bb5c65e4f7db-konnectivity-ca\") pod \"konnectivity-agent-htk8n\" (UID: \"14653c0f-3cc1-4527-b610-bb5c65e4f7db\") " pod="kube-system/konnectivity-agent-htk8n"
Sep 16 12:03:28.869070 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.867356    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e0fe6c2-78d6-4a99-8197-50e62073f615-etc-sysctl-conf\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.869070 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.867494    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc93b327-87ec-4a56-a549-9dbfddce7fc0-cni-binary-copy\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.869070 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.867599    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6180fb10-c394-4bca-97af-1a28211ea33f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cwn4x\" (UID: \"6180fb10-c394-4bca-97af-1a28211ea33f\") " pod="openshift-multus/multus-additional-cni-plugins-cwn4x"
Sep 16 12:03:28.869070 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.868729    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2e0fe6c2-78d6-4a99-8197-50e62073f615-etc-tuned\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.869070 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.869055    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e0fe6c2-78d6-4a99-8197-50e62073f615-tmp\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.869805 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.869460    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/14653c0f-3cc1-4527-b610-bb5c65e4f7db-agent-certs\") pod \"konnectivity-agent-htk8n\" (UID: \"14653c0f-3cc1-4527-b610-bb5c65e4f7db\") " pod="kube-system/konnectivity-agent-htk8n"
Sep 16 12:03:28.873419 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:28.873396    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:03:28.873419 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:28.873422    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:03:28.873578 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:28.873437    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bpfv9 for pod openshift-network-diagnostics/network-check-target-rbck7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:28.873578 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:28.873516    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fdacb7a-b65c-48f5-b4f3-6580109c3dbd-kube-api-access-bpfv9 podName:4fdacb7a-b65c-48f5-b4f3-6580109c3dbd nodeName:}" failed. No retries permitted until 2026-09-16 12:03:29.37349771 +0000 UTC m=+3.074569024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bpfv9" (UniqueName: "kubernetes.io/projected/4fdacb7a-b65c-48f5-b4f3-6580109c3dbd-kube-api-access-bpfv9") pod "network-check-target-rbck7" (UID: "4fdacb7a-b65c-48f5-b4f3-6580109c3dbd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:28.876120 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.876095    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9ght\" (UniqueName: \"kubernetes.io/projected/fc93b327-87ec-4a56-a549-9dbfddce7fc0-kube-api-access-k9ght\") pod \"multus-w64wk\" (UID: \"fc93b327-87ec-4a56-a549-9dbfddce7fc0\") " pod="openshift-multus/multus-w64wk"
Sep 16 12:03:28.876243 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.876221    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjxp5\" (UniqueName: \"kubernetes.io/projected/b38a0789-177d-497c-8d7f-7d7177f5015e-kube-api-access-wjxp5\") pod \"node-resolver-q4hbf\" (UID: \"b38a0789-177d-497c-8d7f-7d7177f5015e\") " pod="openshift-dns/node-resolver-q4hbf"
Sep 16 12:03:28.877057 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.877033    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsbrp\" (UniqueName: \"kubernetes.io/projected/2e0fe6c2-78d6-4a99-8197-50e62073f615-kube-api-access-zsbrp\") pod \"tuned-hxc2w\" (UID: \"2e0fe6c2-78d6-4a99-8197-50e62073f615\") " pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:28.877450 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.877433    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmgkf\" (UniqueName: \"kubernetes.io/projected/6180fb10-c394-4bca-97af-1a28211ea33f-kube-api-access-wmgkf\") pod \"multus-additional-cni-plugins-cwn4x\" (UID: \"6180fb10-c394-4bca-97af-1a28211ea33f\") " pod="openshift-multus/multus-additional-cni-plugins-cwn4x"
Sep 16 12:03:28.959394 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.959356    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ldcrr"
Sep 16 12:03:28.969894 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.969866    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-59h6p"
Sep 16 12:03:28.981264 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.981238    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46"
Sep 16 12:03:28.988299 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.988273    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:28.997096 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:28.997075    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w64wk"
Sep 16 12:03:29.005983 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:29.005963    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cwn4x"
Sep 16 12:03:29.007358 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:29.007337    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 12:03:29.014985 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:29.014960    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q4hbf"
Sep 16 12:03:29.024899 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:29.024881    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hxc2w"
Sep 16 12:03:29.029844 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:29.029823    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-htk8n"
Sep 16 12:03:29.270829 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:29.270741    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs\") pod \"network-metrics-daemon-7cn45\" (UID: \"5441211d-70c6-473a-ac90-417bd7aa57ec\") " pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:29.271010 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:29.270925    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:29.271010 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:29.271009    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs podName:5441211d-70c6-473a-ac90-417bd7aa57ec nodeName:}" failed. No retries permitted until 2026-09-16 12:03:30.27098775 +0000 UTC m=+3.972059069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs") pod "network-metrics-daemon-7cn45" (UID: "5441211d-70c6-473a-ac90-417bd7aa57ec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:29.371557 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:29.371518    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fbbe06db-6359-4c0d-aa65-d07fa8245950-original-pull-secret\") pod \"global-pull-secret-syncer-qgx6n\" (UID: \"fbbe06db-6359-4c0d-aa65-d07fa8245950\") " pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:29.371788 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:29.371674    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:29.371788 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:29.371766    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbbe06db-6359-4c0d-aa65-d07fa8245950-original-pull-secret podName:fbbe06db-6359-4c0d-aa65-d07fa8245950 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:30.371746071 +0000 UTC m=+4.072817377 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fbbe06db-6359-4c0d-aa65-d07fa8245950-original-pull-secret") pod "global-pull-secret-syncer-qgx6n" (UID: "fbbe06db-6359-4c0d-aa65-d07fa8245950") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:29.472510 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:29.472473    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpfv9\" (UniqueName: \"kubernetes.io/projected/4fdacb7a-b65c-48f5-b4f3-6580109c3dbd-kube-api-access-bpfv9\") pod \"network-check-target-rbck7\" (UID: \"4fdacb7a-b65c-48f5-b4f3-6580109c3dbd\") " pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:03:29.472664 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:29.472612    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:03:29.472664 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:29.472632    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:03:29.472664 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:29.472643    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bpfv9 for pod openshift-network-diagnostics/network-check-target-rbck7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:29.472824 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:29.472709    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fdacb7a-b65c-48f5-b4f3-6580109c3dbd-kube-api-access-bpfv9 podName:4fdacb7a-b65c-48f5-b4f3-6580109c3dbd nodeName:}" failed. No retries permitted until 2026-09-16 12:03:30.472679805 +0000 UTC m=+4.173751115 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bpfv9" (UniqueName: "kubernetes.io/projected/4fdacb7a-b65c-48f5-b4f3-6580109c3dbd-kube-api-access-bpfv9") pod "network-check-target-rbck7" (UID: "4fdacb7a-b65c-48f5-b4f3-6580109c3dbd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:29.713066 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:29.713000    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 11:58:29 +0000 UTC" deadline="2028-03-11 23:34:09.99909402 +0000 UTC"
Sep 16 12:03:29.713066 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:29.713051    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13019h30m40.286051428s"
Sep 16 12:03:29.869091 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:29.869064    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6180fb10_c394_4bca_97af_1a28211ea33f.slice/crio-7e102f08b2ddae9aab536038c326db4675284cada64d057b715ba237b699ed5e WatchSource:0}: Error finding container 7e102f08b2ddae9aab536038c326db4675284cada64d057b715ba237b699ed5e: Status 404 returned error can't find the container with id 7e102f08b2ddae9aab536038c326db4675284cada64d057b715ba237b699ed5e
Sep 16 12:03:29.870284 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:29.870258    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cbf941a_3e65_4213_8ab4_fa25f4a27200.slice/crio-762b00e0c14fb66008589c02cc7d9e011427035eb3b203849aa9d8ea24817771 WatchSource:0}: Error finding container 762b00e0c14fb66008589c02cc7d9e011427035eb3b203849aa9d8ea24817771: Status 404 returned error can't find the container with id 762b00e0c14fb66008589c02cc7d9e011427035eb3b203849aa9d8ea24817771
Sep 16 12:03:29.874874 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:29.874769    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14653c0f_3cc1_4527_b610_bb5c65e4f7db.slice/crio-3ac747ff4e3c374cdfd5e58b4fb991225e58e97b9330587039b40fc44239deb9 WatchSource:0}: Error finding container 3ac747ff4e3c374cdfd5e58b4fb991225e58e97b9330587039b40fc44239deb9: Status 404 returned error can't find the container with id 3ac747ff4e3c374cdfd5e58b4fb991225e58e97b9330587039b40fc44239deb9
Sep 16 12:03:29.876041 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:29.875996    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390dfac1_62d4_4a3f_8586_4bca1a499d37.slice/crio-f7194b5bcf7b225a84818cbf6f0133be57f19915fc90a30c8587be9a74aa661f WatchSource:0}: Error finding container f7194b5bcf7b225a84818cbf6f0133be57f19915fc90a30c8587be9a74aa661f: Status 404 returned error can't find the container with id f7194b5bcf7b225a84818cbf6f0133be57f19915fc90a30c8587be9a74aa661f
Sep 16 12:03:29.879184 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:29.878994    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc93b327_87ec_4a56_a549_9dbfddce7fc0.slice/crio-e2eca5fddd156dec97b6e5b152ad7d113eb8c320ef793434d975bfad0ce47f6a WatchSource:0}: Error finding container e2eca5fddd156dec97b6e5b152ad7d113eb8c320ef793434d975bfad0ce47f6a: Status 404 returned error can't find the container with id e2eca5fddd156dec97b6e5b152ad7d113eb8c320ef793434d975bfad0ce47f6a
Sep 16 12:03:29.884019 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:03:29.883990    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod977b895b_048d_42e4_854a_fe9d395f0205.slice/crio-61ea97bea6db7084767a728f27eaff2a5fc143b17cba0a99b36996ca642f892a WatchSource:0}: Error finding container 61ea97bea6db7084767a728f27eaff2a5fc143b17cba0a99b36996ca642f892a: Status 404 returned error can't find the container with id 61ea97bea6db7084767a728f27eaff2a5fc143b17cba0a99b36996ca642f892a
Sep 16 12:03:30.283087 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:30.282805    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs\") pod \"network-metrics-daemon-7cn45\" (UID: \"5441211d-70c6-473a-ac90-417bd7aa57ec\") " pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:30.283087 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:30.282966    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:30.283087 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:30.283026    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs podName:5441211d-70c6-473a-ac90-417bd7aa57ec nodeName:}" failed. No retries permitted until 2026-09-16 12:03:32.283010628 +0000 UTC m=+5.984081948 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs") pod "network-metrics-daemon-7cn45" (UID: "5441211d-70c6-473a-ac90-417bd7aa57ec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:30.383758 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:30.383694    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fbbe06db-6359-4c0d-aa65-d07fa8245950-original-pull-secret\") pod \"global-pull-secret-syncer-qgx6n\" (UID: \"fbbe06db-6359-4c0d-aa65-d07fa8245950\") " pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:30.383957 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:30.383845    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:30.383957 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:30.383922    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbbe06db-6359-4c0d-aa65-d07fa8245950-original-pull-secret podName:fbbe06db-6359-4c0d-aa65-d07fa8245950 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:32.383901735 +0000 UTC m=+6.084973040 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fbbe06db-6359-4c0d-aa65-d07fa8245950-original-pull-secret") pod "global-pull-secret-syncer-qgx6n" (UID: "fbbe06db-6359-4c0d-aa65-d07fa8245950") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:30.484456 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:30.484413    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpfv9\" (UniqueName: \"kubernetes.io/projected/4fdacb7a-b65c-48f5-b4f3-6580109c3dbd-kube-api-access-bpfv9\") pod \"network-check-target-rbck7\" (UID: \"4fdacb7a-b65c-48f5-b4f3-6580109c3dbd\") " pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:03:30.484648 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:30.484585    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:03:30.484648 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:30.484606    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:03:30.484648 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:30.484618    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bpfv9 for pod openshift-network-diagnostics/network-check-target-rbck7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:30.484854 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:30.484681    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fdacb7a-b65c-48f5-b4f3-6580109c3dbd-kube-api-access-bpfv9 podName:4fdacb7a-b65c-48f5-b4f3-6580109c3dbd nodeName:}" failed. No retries permitted until 2026-09-16 12:03:32.484661698 +0000 UTC m=+6.185733004 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bpfv9" (UniqueName: "kubernetes.io/projected/4fdacb7a-b65c-48f5-b4f3-6580109c3dbd-kube-api-access-bpfv9") pod "network-check-target-rbck7" (UID: "4fdacb7a-b65c-48f5-b4f3-6580109c3dbd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:30.714529 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:30.714182    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 11:58:29 +0000 UTC" deadline="2028-03-29 22:24:20.275112851 +0000 UTC"
Sep 16 12:03:30.714529 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:30.714253    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13450h20m49.56086489s"
Sep 16 12:03:30.775901 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:30.775866    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:30.776084 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:30.776019    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7cn45" podUID="5441211d-70c6-473a-ac90-417bd7aa57ec"
Sep 16 12:03:30.776468 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:30.776445    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:30.776574 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:30.776551    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgx6n" podUID="fbbe06db-6359-4c0d-aa65-d07fa8245950"
Sep 16 12:03:30.776670 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:30.776654    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:03:30.776775 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:30.776756    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbck7" podUID="4fdacb7a-b65c-48f5-b4f3-6580109c3dbd"
Sep 16 12:03:30.800014 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:30.799959    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cwn4x" event={"ID":"6180fb10-c394-4bca-97af-1a28211ea33f","Type":"ContainerStarted","Data":"7e102f08b2ddae9aab536038c326db4675284cada64d057b715ba237b699ed5e"}
Sep 16 12:03:30.814198 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:30.814161    2579 generic.go:356] "Generic (PLEG): container finished" podID="e411ae70a2a6889284813c1948077cd9" containerID="ae567055b75a154af94f0aa215084bda7cb3e72de72da6086015faae2d61490c" exitCode=0
Sep 16 12:03:30.814345 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:30.814297    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-30.ec2.internal" event={"ID":"e411ae70a2a6889284813c1948077cd9","Type":"ContainerDied","Data":"ae567055b75a154af94f0aa215084bda7cb3e72de72da6086015faae2d61490c"}
Sep 16 12:03:30.818448 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:30.818373    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hxc2w" event={"ID":"2e0fe6c2-78d6-4a99-8197-50e62073f615","Type":"ContainerStarted","Data":"81c41dc078cb2833c0b62a00115933f733f1559c4df76e1e54ae2b593e98e5d9"}
Sep 16 12:03:30.829457 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:30.829425    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ldcrr" event={"ID":"a6a91efe-a2f3-4829-9777-1ea089f08030","Type":"ContainerStarted","Data":"d52a23ac9c0253c6e02d19a484b5552ad91e70269d4e846911634feefefc9627"}
Sep 16 12:03:30.839882 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:30.839852    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-htk8n" event={"ID":"14653c0f-3cc1-4527-b610-bb5c65e4f7db","Type":"ContainerStarted","Data":"3ac747ff4e3c374cdfd5e58b4fb991225e58e97b9330587039b40fc44239deb9"}
Sep 16 12:03:30.860966 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:30.860934    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-30.ec2.internal" event={"ID":"3d94b98261326c611e79bccb709825c5","Type":"ContainerStarted","Data":"54ccd3df899923b180352fa251468a41584a19e2f452d3b0fd4cc0bc74a6b449"}
Sep 16 12:03:30.891841 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:30.891804    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-59h6p" event={"ID":"390dfac1-62d4-4a3f-8586-4bca1a499d37","Type":"ContainerStarted","Data":"f7194b5bcf7b225a84818cbf6f0133be57f19915fc90a30c8587be9a74aa661f"}
Sep 16 12:03:30.899978 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:30.899943    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46" event={"ID":"977b895b-048d-42e4-854a-fe9d395f0205","Type":"ContainerStarted","Data":"61ea97bea6db7084767a728f27eaff2a5fc143b17cba0a99b36996ca642f892a"}
Sep 16 12:03:30.909942 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:30.909859    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q4hbf" event={"ID":"b38a0789-177d-497c-8d7f-7d7177f5015e","Type":"ContainerStarted","Data":"1f5c384099c1922578e13072e65cc6430e1b0797bed862a10222598ad0097b54"}
Sep 16 12:03:30.919355 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:30.919325    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w64wk" event={"ID":"fc93b327-87ec-4a56-a549-9dbfddce7fc0","Type":"ContainerStarted","Data":"e2eca5fddd156dec97b6e5b152ad7d113eb8c320ef793434d975bfad0ce47f6a"}
Sep 16 12:03:30.935187 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:30.935145    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg" event={"ID":"6cbf941a-3e65-4213-8ab4-fa25f4a27200","Type":"ContainerStarted","Data":"762b00e0c14fb66008589c02cc7d9e011427035eb3b203849aa9d8ea24817771"}
Sep 16 12:03:31.967012 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:31.966869    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-30.ec2.internal" event={"ID":"e411ae70a2a6889284813c1948077cd9","Type":"ContainerStarted","Data":"5664a79aebba35659da9ee15542c29be1c6ec8b2e0b79e0baff80ecc17790d13"}
Sep 16 12:03:31.985511 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:31.985378    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-30.ec2.internal" podStartSLOduration=3.985352903 podStartE2EDuration="3.985352903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:30.879071231 +0000 UTC m=+4.580142561" watchObservedRunningTime="2026-09-16 12:03:31.985352903 +0000 UTC m=+5.686424250"
Sep 16 12:03:31.985796 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:31.985727    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-30.ec2.internal" podStartSLOduration=3.985713779 podStartE2EDuration="3.985713779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:31.983371534 +0000 UTC m=+5.684442881" watchObservedRunningTime="2026-09-16 12:03:31.985713779 +0000 UTC m=+5.686785112"
Sep 16 12:03:32.300272 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:32.299550    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs\") pod \"network-metrics-daemon-7cn45\" (UID: \"5441211d-70c6-473a-ac90-417bd7aa57ec\") " pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:32.300272 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:32.299767    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:32.300272 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:32.299833    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs podName:5441211d-70c6-473a-ac90-417bd7aa57ec nodeName:}" failed. No retries permitted until 2026-09-16 12:03:36.29981358 +0000 UTC m=+10.000884889 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs") pod "network-metrics-daemon-7cn45" (UID: "5441211d-70c6-473a-ac90-417bd7aa57ec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:32.401013 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:32.400969    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fbbe06db-6359-4c0d-aa65-d07fa8245950-original-pull-secret\") pod \"global-pull-secret-syncer-qgx6n\" (UID: \"fbbe06db-6359-4c0d-aa65-d07fa8245950\") " pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:32.401208 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:32.401166    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:32.401301 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:32.401233    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbbe06db-6359-4c0d-aa65-d07fa8245950-original-pull-secret podName:fbbe06db-6359-4c0d-aa65-d07fa8245950 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:36.401214818 +0000 UTC m=+10.102286140 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fbbe06db-6359-4c0d-aa65-d07fa8245950-original-pull-secret") pod "global-pull-secret-syncer-qgx6n" (UID: "fbbe06db-6359-4c0d-aa65-d07fa8245950") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:32.503004 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:32.502266    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpfv9\" (UniqueName: \"kubernetes.io/projected/4fdacb7a-b65c-48f5-b4f3-6580109c3dbd-kube-api-access-bpfv9\") pod \"network-check-target-rbck7\" (UID: \"4fdacb7a-b65c-48f5-b4f3-6580109c3dbd\") " pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:03:32.503004 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:32.502485    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:03:32.503004 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:32.502512    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:03:32.503004 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:32.502527    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bpfv9 for pod openshift-network-diagnostics/network-check-target-rbck7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:32.503004 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:32.502592    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fdacb7a-b65c-48f5-b4f3-6580109c3dbd-kube-api-access-bpfv9 podName:4fdacb7a-b65c-48f5-b4f3-6580109c3dbd nodeName:}" failed. No retries permitted until 2026-09-16 12:03:36.502572374 +0000 UTC m=+10.203643701 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bpfv9" (UniqueName: "kubernetes.io/projected/4fdacb7a-b65c-48f5-b4f3-6580109c3dbd-kube-api-access-bpfv9") pod "network-check-target-rbck7" (UID: "4fdacb7a-b65c-48f5-b4f3-6580109c3dbd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:32.777473 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:32.777436    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:32.777659 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:32.777594    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7cn45" podUID="5441211d-70c6-473a-ac90-417bd7aa57ec"
Sep 16 12:03:32.778345 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:32.778005    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:32.778345 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:32.778120    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgx6n" podUID="fbbe06db-6359-4c0d-aa65-d07fa8245950"
Sep 16 12:03:32.778345 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:32.778174    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:03:32.778345 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:32.778238    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbck7" podUID="4fdacb7a-b65c-48f5-b4f3-6580109c3dbd"
Sep 16 12:03:34.775336 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:34.775299    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:34.775842 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:34.775463    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7cn45" podUID="5441211d-70c6-473a-ac90-417bd7aa57ec"
Sep 16 12:03:34.775947 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:34.775928    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:34.776055 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:34.776031    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgx6n" podUID="fbbe06db-6359-4c0d-aa65-d07fa8245950"
Sep 16 12:03:34.776147 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:34.776131    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:03:34.776234 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:34.776215    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbck7" podUID="4fdacb7a-b65c-48f5-b4f3-6580109c3dbd"
Sep 16 12:03:36.337614 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:36.337569    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs\") pod \"network-metrics-daemon-7cn45\" (UID: \"5441211d-70c6-473a-ac90-417bd7aa57ec\") " pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:36.338206 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:36.337735    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:36.338206 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:36.337824    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs podName:5441211d-70c6-473a-ac90-417bd7aa57ec nodeName:}" failed. No retries permitted until 2026-09-16 12:03:44.337800685 +0000 UTC m=+18.038872005 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs") pod "network-metrics-daemon-7cn45" (UID: "5441211d-70c6-473a-ac90-417bd7aa57ec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:36.438974 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:36.438803    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fbbe06db-6359-4c0d-aa65-d07fa8245950-original-pull-secret\") pod \"global-pull-secret-syncer-qgx6n\" (UID: \"fbbe06db-6359-4c0d-aa65-d07fa8245950\") " pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:36.438974 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:36.438966    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:36.439273 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:36.439029    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbbe06db-6359-4c0d-aa65-d07fa8245950-original-pull-secret podName:fbbe06db-6359-4c0d-aa65-d07fa8245950 nodeName:}" failed. No retries permitted until 2026-09-16 12:03:44.439011177 +0000 UTC m=+18.140082483 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fbbe06db-6359-4c0d-aa65-d07fa8245950-original-pull-secret") pod "global-pull-secret-syncer-qgx6n" (UID: "fbbe06db-6359-4c0d-aa65-d07fa8245950") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:36.539218 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:36.539178    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpfv9\" (UniqueName: \"kubernetes.io/projected/4fdacb7a-b65c-48f5-b4f3-6580109c3dbd-kube-api-access-bpfv9\") pod \"network-check-target-rbck7\" (UID: \"4fdacb7a-b65c-48f5-b4f3-6580109c3dbd\") " pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:03:36.539388 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:36.539370    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:03:36.539435 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:36.539395    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:03:36.539435 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:36.539409    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bpfv9 for pod openshift-network-diagnostics/network-check-target-rbck7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:36.539590 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:36.539466    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fdacb7a-b65c-48f5-b4f3-6580109c3dbd-kube-api-access-bpfv9 podName:4fdacb7a-b65c-48f5-b4f3-6580109c3dbd nodeName:}" failed. No retries permitted until 2026-09-16 12:03:44.539446667 +0000 UTC m=+18.240517976 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bpfv9" (UniqueName: "kubernetes.io/projected/4fdacb7a-b65c-48f5-b4f3-6580109c3dbd-kube-api-access-bpfv9") pod "network-check-target-rbck7" (UID: "4fdacb7a-b65c-48f5-b4f3-6580109c3dbd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:36.775917 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:36.775753    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:03:36.775917 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:36.775866    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbck7" podUID="4fdacb7a-b65c-48f5-b4f3-6580109c3dbd"
Sep 16 12:03:36.776139 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:36.775926    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:36.776139 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:36.775984    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgx6n" podUID="fbbe06db-6359-4c0d-aa65-d07fa8245950"
Sep 16 12:03:36.776494 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:36.776319    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:36.776494 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:36.776453    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7cn45" podUID="5441211d-70c6-473a-ac90-417bd7aa57ec"
Sep 16 12:03:38.775135 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:38.774522    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:38.775135 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:38.774569    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:03:38.775135 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:38.774522    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:38.775724 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:38.775515    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgx6n" podUID="fbbe06db-6359-4c0d-aa65-d07fa8245950"
Sep 16 12:03:38.775724 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:38.775696    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7cn45" podUID="5441211d-70c6-473a-ac90-417bd7aa57ec"
Sep 16 12:03:38.775850 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:38.775774    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbck7" podUID="4fdacb7a-b65c-48f5-b4f3-6580109c3dbd"
Sep 16 12:03:40.774739 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:40.774707    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:40.775158 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:40.774714    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:03:40.775158 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:40.774808    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgx6n" podUID="fbbe06db-6359-4c0d-aa65-d07fa8245950"
Sep 16 12:03:40.775158 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:40.774726    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:40.775158 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:40.774884    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbck7" podUID="4fdacb7a-b65c-48f5-b4f3-6580109c3dbd"
Sep 16 12:03:40.775158 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:40.774949    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7cn45" podUID="5441211d-70c6-473a-ac90-417bd7aa57ec"
Sep 16 12:03:42.774810 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:42.774769    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:42.775232 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:42.774835    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:03:42.775232 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:42.774843    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:42.775232 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:42.774949    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7cn45" podUID="5441211d-70c6-473a-ac90-417bd7aa57ec"
Sep 16 12:03:42.775232 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:42.775054    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbck7" podUID="4fdacb7a-b65c-48f5-b4f3-6580109c3dbd"
Sep 16 12:03:42.775232 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:42.775133    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgx6n" podUID="fbbe06db-6359-4c0d-aa65-d07fa8245950"
Sep 16 12:03:44.402173 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:44.402138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs\") pod \"network-metrics-daemon-7cn45\" (UID: \"5441211d-70c6-473a-ac90-417bd7aa57ec\") " pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:44.402649 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:44.402269    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:44.402649 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:44.402341    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs podName:5441211d-70c6-473a-ac90-417bd7aa57ec nodeName:}" failed. No retries permitted until 2026-09-16 12:04:00.402321083 +0000 UTC m=+34.103392397 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs") pod "network-metrics-daemon-7cn45" (UID: "5441211d-70c6-473a-ac90-417bd7aa57ec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:03:44.503347 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:44.503315    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fbbe06db-6359-4c0d-aa65-d07fa8245950-original-pull-secret\") pod \"global-pull-secret-syncer-qgx6n\" (UID: \"fbbe06db-6359-4c0d-aa65-d07fa8245950\") " pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:44.503531 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:44.503454    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:44.503531 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:44.503523    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbbe06db-6359-4c0d-aa65-d07fa8245950-original-pull-secret podName:fbbe06db-6359-4c0d-aa65-d07fa8245950 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:00.503505536 +0000 UTC m=+34.204576853 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fbbe06db-6359-4c0d-aa65-d07fa8245950-original-pull-secret") pod "global-pull-secret-syncer-qgx6n" (UID: "fbbe06db-6359-4c0d-aa65-d07fa8245950") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:03:44.604787 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:44.604753    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpfv9\" (UniqueName: \"kubernetes.io/projected/4fdacb7a-b65c-48f5-b4f3-6580109c3dbd-kube-api-access-bpfv9\") pod \"network-check-target-rbck7\" (UID: \"4fdacb7a-b65c-48f5-b4f3-6580109c3dbd\") " pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:03:44.604948 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:44.604913    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:03:44.604948 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:44.604935    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:03:44.604948 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:44.604946    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bpfv9 for pod openshift-network-diagnostics/network-check-target-rbck7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:44.605048 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:44.604994    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fdacb7a-b65c-48f5-b4f3-6580109c3dbd-kube-api-access-bpfv9 podName:4fdacb7a-b65c-48f5-b4f3-6580109c3dbd nodeName:}" failed. No retries permitted until 2026-09-16 12:04:00.604981958 +0000 UTC m=+34.306053276 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bpfv9" (UniqueName: "kubernetes.io/projected/4fdacb7a-b65c-48f5-b4f3-6580109c3dbd-kube-api-access-bpfv9") pod "network-check-target-rbck7" (UID: "4fdacb7a-b65c-48f5-b4f3-6580109c3dbd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:03:44.774527 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:44.774453    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:44.774527 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:44.774470    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:03:44.774527 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:44.774470    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:44.774768 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:44.774607    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7cn45" podUID="5441211d-70c6-473a-ac90-417bd7aa57ec"
Sep 16 12:03:44.774768 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:44.774724    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbck7" podUID="4fdacb7a-b65c-48f5-b4f3-6580109c3dbd"
Sep 16 12:03:44.774845 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:44.774796    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgx6n" podUID="fbbe06db-6359-4c0d-aa65-d07fa8245950"
Sep 16 12:03:46.774761 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:46.774717    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:46.775213 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:46.774828    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:46.775213 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:46.774853    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7cn45" podUID="5441211d-70c6-473a-ac90-417bd7aa57ec"
Sep 16 12:03:46.775213 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:46.774924    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgx6n" podUID="fbbe06db-6359-4c0d-aa65-d07fa8245950"
Sep 16 12:03:46.775213 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:46.774973    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:03:46.775213 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:46.775034    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbck7" podUID="4fdacb7a-b65c-48f5-b4f3-6580109c3dbd"
Sep 16 12:03:48.774278 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:48.773968    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:48.774846 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:48.773968    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:03:48.774846 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:48.774369    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgx6n" podUID="fbbe06db-6359-4c0d-aa65-d07fa8245950"
Sep 16 12:03:48.774846 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:48.773976    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:48.774846 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:48.774467    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbck7" podUID="4fdacb7a-b65c-48f5-b4f3-6580109c3dbd"
Sep 16 12:03:48.774846 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:48.774562    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7cn45" podUID="5441211d-70c6-473a-ac90-417bd7aa57ec"
Sep 16 12:03:49.001828 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:49.001794    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46" event={"ID":"977b895b-048d-42e4-854a-fe9d395f0205","Type":"ContainerStarted","Data":"92ee07ffd702490797fb7c530450c47d74ec2a775401d47d36e62db3d3f83ed1"}
Sep 16 12:03:49.003413 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:49.003376    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q4hbf" event={"ID":"b38a0789-177d-497c-8d7f-7d7177f5015e","Type":"ContainerStarted","Data":"d436c070343d266ae52ef0c6c5e9ee3bda0db89243a91787e0c6defba91da6be"}
Sep 16 12:03:49.006583 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:49.006552    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w64wk" event={"ID":"fc93b327-87ec-4a56-a549-9dbfddce7fc0","Type":"ContainerStarted","Data":"9e5c75009606a0696678ae1a4ccf4cf49ede41c4653c2dad8fc1443eefb41762"}
Sep 16 12:03:49.009535 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:49.009515    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg" event={"ID":"6cbf941a-3e65-4213-8ab4-fa25f4a27200","Type":"ContainerStarted","Data":"71645b725bd3facc7cf81142c50e2c2329796ce0a1fc1bd288e654607bd48eb0"}
Sep 16 12:03:49.009634 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:49.009576    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg" event={"ID":"6cbf941a-3e65-4213-8ab4-fa25f4a27200","Type":"ContainerStarted","Data":"06355443a60b0dc90ee1046812fe2e53ba79f534865309f0066a79d2a9b124a9"}
Sep 16 12:03:49.009634 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:49.009606    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg" event={"ID":"6cbf941a-3e65-4213-8ab4-fa25f4a27200","Type":"ContainerStarted","Data":"c353b2c9e3490fdb94d950e7d8ed0000d6cdda3a0a1d35f44b0c5db1c2e26773"}
Sep 16 12:03:49.009783 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:49.009646    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg" event={"ID":"6cbf941a-3e65-4213-8ab4-fa25f4a27200","Type":"ContainerStarted","Data":"8a8855aa29c2dedd3f19c24b47e556237240f640e3a97929bf89919675b32035"}
Sep 16 12:03:49.009783 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:49.009692    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg" event={"ID":"6cbf941a-3e65-4213-8ab4-fa25f4a27200","Type":"ContainerStarted","Data":"fbac4d206c08b6cf05017e9756df6fd984af3936c974660cedb6787134cf5b0e"}
Sep 16 12:03:49.009783 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:49.009726    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg" event={"ID":"6cbf941a-3e65-4213-8ab4-fa25f4a27200","Type":"ContainerStarted","Data":"0177510067de1de8d0012d8e8c7a86abaac101f27edd00e81d9a26f78a91ad48"}
Sep 16 12:03:49.011182 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:49.011157    2579 generic.go:356] "Generic (PLEG): container finished" podID="6180fb10-c394-4bca-97af-1a28211ea33f" containerID="7b8f1bde359bb68efb94ea8793672ffa91f0e9348bb056d616a972e8d5e35909" exitCode=0
Sep 16 12:03:49.011273 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:49.011229    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cwn4x" event={"ID":"6180fb10-c394-4bca-97af-1a28211ea33f","Type":"ContainerDied","Data":"7b8f1bde359bb68efb94ea8793672ffa91f0e9348bb056d616a972e8d5e35909"}
Sep 16 12:03:49.012911 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:49.012886    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hxc2w" event={"ID":"2e0fe6c2-78d6-4a99-8197-50e62073f615","Type":"ContainerStarted","Data":"1a4a9affd1c53b272faac7f7f10427002e7c0ffc6964c6d4e6e8e7798258ebe5"}
Sep 16 12:03:49.014328 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:49.014309    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ldcrr" event={"ID":"a6a91efe-a2f3-4829-9777-1ea089f08030","Type":"ContainerStarted","Data":"4a3f5821c3fb48aa82b7740006643104412156e7c8f93c2f16b7eae711e441c1"}
Sep 16 12:03:49.015619 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:49.015602    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-htk8n" event={"ID":"14653c0f-3cc1-4527-b610-bb5c65e4f7db","Type":"ContainerStarted","Data":"739493c58d7c6dc6897fa345dcf15b65b8c873dc268666e2f43cf45bcfca6e37"}
Sep 16 12:03:49.017941 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:49.017904    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-q4hbf" podStartSLOduration=3.598254133 podStartE2EDuration="22.017891569s" totalImagesPullingTime="18.419637436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:49.017379449 +0000 UTC m=+22.718450788" watchObservedRunningTime="2026-09-16 12:03:49.017891569 +0000 UTC m=+22.718962896"
Sep 16 12:03:49.051729 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:49.051645    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hxc2w" podStartSLOduration=3.602550136 podStartE2EDuration="22.05162899s" totalImagesPullingTime="18.449078854s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:49.051038962 +0000 UTC m=+22.752110288" watchObservedRunningTime="2026-09-16 12:03:49.05162899 +0000 UTC m=+22.752700313"
Sep 16 12:03:49.064376 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:49.064320    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ldcrr" podStartSLOduration=3.616430426 podStartE2EDuration="22.064298674s" totalImagesPullingTime="18.447868248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:49.063811459 +0000 UTC m=+22.764882785" watchObservedRunningTime="2026-09-16 12:03:49.064298674 +0000 UTC m=+22.765370000"
Sep 16 12:03:49.080768 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:49.080711    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-w64wk" podStartSLOduration=3.583446043 podStartE2EDuration="22.080689658s" totalImagesPullingTime="18.497243615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:49.080217603 +0000 UTC m=+22.781288930" watchObservedRunningTime="2026-09-16 12:03:49.080689658 +0000 UTC m=+22.781760990"
Sep 16 12:03:49.095886 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:49.095834    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-htk8n" podStartSLOduration=7.444197026 podStartE2EDuration="22.095818775s" totalImagesPullingTime="14.651621749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:49.095542564 +0000 UTC m=+22.796613882" watchObservedRunningTime="2026-09-16 12:03:49.095818775 +0000 UTC m=+22.796890101"
Sep 16 12:03:49.827146 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:49.827118    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 12:03:50.019181 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:50.019059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-59h6p" event={"ID":"390dfac1-62d4-4a3f-8586-4bca1a499d37","Type":"ContainerStarted","Data":"f5ab1b585454c2c7ba01b69bbcc5bdb78ebce72a0f42d248361dc1898b911f87"}
Sep 16 12:03:50.021453 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:50.021398    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46" event={"ID":"977b895b-048d-42e4-854a-fe9d395f0205","Type":"ContainerStarted","Data":"e37642d1e72900c843c521b6a4bd4e606dd1f6e31851dfc7d53b3f571a139386"}
Sep 16 12:03:50.034545 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:50.034502    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-59h6p" podStartSLOduration=8.385269516 podStartE2EDuration="23.034484722s" totalImagesPullingTime="14.649215206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:50.034321353 +0000 UTC m=+23.735392683" watchObservedRunningTime="2026-09-16 12:03:50.034484722 +0000 UTC m=+23.735556049"
Sep 16 12:03:50.715627 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:50.715510    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T12:03:49.827140784Z","UUID":"f0396a0f-173f-4138-8d39-f0d95fc47b68","Handler":null,"Name":"","Endpoint":""}
Sep 16 12:03:50.717017 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:50.716999    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 12:03:50.717138 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:50.717038    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 12:03:50.726293 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:50.726270    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-htk8n"
Sep 16 12:03:50.726796 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:50.726773    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-htk8n"
Sep 16 12:03:50.774476 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:50.774447    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:50.774476 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:50.774478    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:03:50.774741 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:50.774447    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:50.774741 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:50.774590    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7cn45" podUID="5441211d-70c6-473a-ac90-417bd7aa57ec"
Sep 16 12:03:50.774841 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:50.774756    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgx6n" podUID="fbbe06db-6359-4c0d-aa65-d07fa8245950"
Sep 16 12:03:50.774895 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:50.774859    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbck7" podUID="4fdacb7a-b65c-48f5-b4f3-6580109c3dbd"
Sep 16 12:03:51.026758 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:51.026728    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46" event={"ID":"977b895b-048d-42e4-854a-fe9d395f0205","Type":"ContainerStarted","Data":"9e7ea381a121eac0a8beda179d47b193cde05e60892edcf9fff4193be4b2ec77"}
Sep 16 12:03:51.027381 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:51.027354    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-htk8n"
Sep 16 12:03:51.028363 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:51.028343    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-htk8n"
Sep 16 12:03:51.045830 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:51.045776    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ntn46" podStartSLOduration=3.4932505369999998 podStartE2EDuration="24.045754904s" totalImagesPullingTime="20.552504367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:51.045727819 +0000 UTC m=+24.746799147" watchObservedRunningTime="2026-09-16 12:03:51.045754904 +0000 UTC m=+24.746826223"
Sep 16 12:03:52.032688 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:52.032458    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg" event={"ID":"6cbf941a-3e65-4213-8ab4-fa25f4a27200","Type":"ContainerStarted","Data":"7d9c6952626f4324ef85ae59dd0f27f227ef01c542208cdcb529b69c1a7a9caa"}
Sep 16 12:03:52.774667 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:52.774621    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:03:52.774889 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:52.774621    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:52.774889 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:52.774770    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbck7" podUID="4fdacb7a-b65c-48f5-b4f3-6580109c3dbd"
Sep 16 12:03:52.774889 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:52.774780    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:52.774889 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:52.774856    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgx6n" podUID="fbbe06db-6359-4c0d-aa65-d07fa8245950"
Sep 16 12:03:52.775074 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:52.774917    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7cn45" podUID="5441211d-70c6-473a-ac90-417bd7aa57ec"
Sep 16 12:03:54.040889 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:54.040864    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg" event={"ID":"6cbf941a-3e65-4213-8ab4-fa25f4a27200","Type":"ContainerStarted","Data":"3c8ebb423f555bcee6bdb9aad874b4ce0148cf8cadc913688275bf25df5d8d7f"}
Sep 16 12:03:54.041269 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:54.041117    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:54.041269 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:54.041174    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:54.055557 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:54.055536    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:54.066599 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:54.066560    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg" podStartSLOduration=8.56059966 podStartE2EDuration="27.066549387s" totalImagesPullingTime="18.505949727s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:03:54.066382627 +0000 UTC m=+27.767453952" watchObservedRunningTime="2026-09-16 12:03:54.066549387 +0000 UTC m=+27.767620712"
Sep 16 12:03:54.774151 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:54.773978    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:54.774314 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:54.773992    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:03:54.774314 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:54.774226    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgx6n" podUID="fbbe06db-6359-4c0d-aa65-d07fa8245950"
Sep 16 12:03:54.774314 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:54.774010    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:54.774451 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:54.774311    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbck7" podUID="4fdacb7a-b65c-48f5-b4f3-6580109c3dbd"
Sep 16 12:03:54.774451 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:54.774408    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7cn45" podUID="5441211d-70c6-473a-ac90-417bd7aa57ec"
Sep 16 12:03:55.045072 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:55.044984    2579 generic.go:356] "Generic (PLEG): container finished" podID="6180fb10-c394-4bca-97af-1a28211ea33f" containerID="fd8c1d9909e528ba1ba34e022b1e45bb0b73dff1e4358673fbdf4ebafabd9e53" exitCode=0
Sep 16 12:03:55.045509 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:55.045066    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cwn4x" event={"ID":"6180fb10-c394-4bca-97af-1a28211ea33f","Type":"ContainerDied","Data":"fd8c1d9909e528ba1ba34e022b1e45bb0b73dff1e4358673fbdf4ebafabd9e53"}
Sep 16 12:03:55.046207 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:55.046162    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:55.062056 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:55.062038    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:03:55.843575 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:55.843536    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qgx6n"]
Sep 16 12:03:55.843760 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:55.843701    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:55.843823 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:55.843806    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgx6n" podUID="fbbe06db-6359-4c0d-aa65-d07fa8245950"
Sep 16 12:03:55.844226 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:55.844206    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rbck7"]
Sep 16 12:03:55.844312 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:55.844285    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:03:55.844374 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:55.844357    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbck7" podUID="4fdacb7a-b65c-48f5-b4f3-6580109c3dbd"
Sep 16 12:03:55.847132 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:55.847111    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7cn45"]
Sep 16 12:03:55.847227 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:55.847215    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:55.847341 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:55.847324    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7cn45" podUID="5441211d-70c6-473a-ac90-417bd7aa57ec"
Sep 16 12:03:57.051099 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:57.051069    2579 generic.go:356] "Generic (PLEG): container finished" podID="6180fb10-c394-4bca-97af-1a28211ea33f" containerID="978e11ce8bc7df95fa5e6df4bdd765c423cbdde0de67249f62bc14fd9a52fa7c" exitCode=0
Sep 16 12:03:57.051874 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:57.051156    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cwn4x" event={"ID":"6180fb10-c394-4bca-97af-1a28211ea33f","Type":"ContainerDied","Data":"978e11ce8bc7df95fa5e6df4bdd765c423cbdde0de67249f62bc14fd9a52fa7c"}
Sep 16 12:03:57.774864 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:57.774679    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:03:57.775317 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:57.774719    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:57.775317 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:57.774951    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbck7" podUID="4fdacb7a-b65c-48f5-b4f3-6580109c3dbd"
Sep 16 12:03:57.775317 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:57.774745    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:57.775317 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:57.775007    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgx6n" podUID="fbbe06db-6359-4c0d-aa65-d07fa8245950"
Sep 16 12:03:57.775317 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:57.775084    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7cn45" podUID="5441211d-70c6-473a-ac90-417bd7aa57ec"
Sep 16 12:03:59.058739 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:59.058709    2579 generic.go:356] "Generic (PLEG): container finished" podID="6180fb10-c394-4bca-97af-1a28211ea33f" containerID="9a959029600c033999d31d4a0cf59fa17476782d2e41d925df2050da22872844" exitCode=0
Sep 16 12:03:59.059086 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:59.058752    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cwn4x" event={"ID":"6180fb10-c394-4bca-97af-1a28211ea33f","Type":"ContainerDied","Data":"9a959029600c033999d31d4a0cf59fa17476782d2e41d925df2050da22872844"}
Sep 16 12:03:59.774117 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:59.774088    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:03:59.774330 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:59.774088    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:03:59.774330 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:59.774209    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7cn45" podUID="5441211d-70c6-473a-ac90-417bd7aa57ec"
Sep 16 12:03:59.774330 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:03:59.774088    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:03:59.774330 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:59.774275    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbck7" podUID="4fdacb7a-b65c-48f5-b4f3-6580109c3dbd"
Sep 16 12:03:59.774478 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:03:59.774344    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgx6n" podUID="fbbe06db-6359-4c0d-aa65-d07fa8245950"
Sep 16 12:04:00.418740 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:00.418699    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs\") pod \"network-metrics-daemon-7cn45\" (UID: \"5441211d-70c6-473a-ac90-417bd7aa57ec\") " pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:04:00.419171 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:00.418856    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:04:00.419171 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:00.418936    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs podName:5441211d-70c6-473a-ac90-417bd7aa57ec nodeName:}" failed. No retries permitted until 2026-09-16 12:04:32.418914789 +0000 UTC m=+66.119986103 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs") pod "network-metrics-daemon-7cn45" (UID: "5441211d-70c6-473a-ac90-417bd7aa57ec") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 12:04:00.519646 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:00.519602    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fbbe06db-6359-4c0d-aa65-d07fa8245950-original-pull-secret\") pod \"global-pull-secret-syncer-qgx6n\" (UID: \"fbbe06db-6359-4c0d-aa65-d07fa8245950\") " pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:04:00.519829 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:00.519760    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 12:04:00.519875 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:00.519830    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbbe06db-6359-4c0d-aa65-d07fa8245950-original-pull-secret podName:fbbe06db-6359-4c0d-aa65-d07fa8245950 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:32.519809466 +0000 UTC m=+66.220880772 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fbbe06db-6359-4c0d-aa65-d07fa8245950-original-pull-secret") pod "global-pull-secret-syncer-qgx6n" (UID: "fbbe06db-6359-4c0d-aa65-d07fa8245950") : object "kube-system"/"original-pull-secret" not registered
Sep 16 12:04:00.620498 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:00.620461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpfv9\" (UniqueName: \"kubernetes.io/projected/4fdacb7a-b65c-48f5-b4f3-6580109c3dbd-kube-api-access-bpfv9\") pod \"network-check-target-rbck7\" (UID: \"4fdacb7a-b65c-48f5-b4f3-6580109c3dbd\") " pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:04:00.620701 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:00.620607    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 12:04:00.620701 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:00.620626    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 12:04:00.620701 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:00.620638    2579 projected.go:196] Error preparing data for projected volume kube-api-access-bpfv9 for pod openshift-network-diagnostics/network-check-target-rbck7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:04:00.620873 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:00.620713    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fdacb7a-b65c-48f5-b4f3-6580109c3dbd-kube-api-access-bpfv9 podName:4fdacb7a-b65c-48f5-b4f3-6580109c3dbd nodeName:}" failed. No retries permitted until 2026-09-16 12:04:32.620693887 +0000 UTC m=+66.321765192 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bpfv9" (UniqueName: "kubernetes.io/projected/4fdacb7a-b65c-48f5-b4f3-6580109c3dbd-kube-api-access-bpfv9") pod "network-check-target-rbck7" (UID: "4fdacb7a-b65c-48f5-b4f3-6580109c3dbd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 12:04:01.774916 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:01.774883    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:04:01.775341 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:01.774995    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:04:01.775341 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:01.775012    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:04:01.775341 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:01.775001    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rbck7" podUID="4fdacb7a-b65c-48f5-b4f3-6580109c3dbd"
Sep 16 12:04:01.775341 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:01.775109    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qgx6n" podUID="fbbe06db-6359-4c0d-aa65-d07fa8245950"
Sep 16 12:04:01.775341 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:01.775197    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7cn45" podUID="5441211d-70c6-473a-ac90-417bd7aa57ec"
Sep 16 12:04:02.069476 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.069403    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-30.ec2.internal" event="NodeReady"
Sep 16 12:04:02.069632 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.069551    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 12:04:02.112255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.112223    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mlc2p"]
Sep 16 12:04:02.114959 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.114923    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:04:02.115307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.115206    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4hvw6"]
Sep 16 12:04:02.116849 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.116829    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4hvw6"
Sep 16 12:04:02.117610 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.117540    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 12:04:02.117610 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.117569    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 12:04:02.117610 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.117608    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7d4qr\""
Sep 16 12:04:02.119166 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.119143    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 12:04:02.119269 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.119232    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 12:04:02.119269 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.119232    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nn9m4\""
Sep 16 12:04:02.119562 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.119545    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 12:04:02.128606 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.127731    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mlc2p"]
Sep 16 12:04:02.129267 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.129244    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4hvw6"]
Sep 16 12:04:02.229962 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.229920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-config-volume\") pod \"dns-default-mlc2p\" (UID: \"096c53b4-6072-4b7e-9d0c-f2ffb8e13694\") " pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:04:02.229962 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.229961    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b4372da-bcb1-4e89-8cd1-84642aa32972-cert\") pod \"ingress-canary-4hvw6\" (UID: \"3b4372da-bcb1-4e89-8cd1-84642aa32972\") " pod="openshift-ingress-canary/ingress-canary-4hvw6"
Sep 16 12:04:02.230190 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.229987    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls\") pod \"dns-default-mlc2p\" (UID: \"096c53b4-6072-4b7e-9d0c-f2ffb8e13694\") " pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:04:02.230190 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.230013    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-tmp-dir\") pod \"dns-default-mlc2p\" (UID: \"096c53b4-6072-4b7e-9d0c-f2ffb8e13694\") " pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:04:02.230190 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.230137    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsc8s\" (UniqueName: \"kubernetes.io/projected/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-kube-api-access-bsc8s\") pod \"dns-default-mlc2p\" (UID: \"096c53b4-6072-4b7e-9d0c-f2ffb8e13694\") " pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:04:02.230190 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.230179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv644\" (UniqueName: \"kubernetes.io/projected/3b4372da-bcb1-4e89-8cd1-84642aa32972-kube-api-access-gv644\") pod \"ingress-canary-4hvw6\" (UID: \"3b4372da-bcb1-4e89-8cd1-84642aa32972\") " pod="openshift-ingress-canary/ingress-canary-4hvw6"
Sep 16 12:04:02.331507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.331470    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-tmp-dir\") pod \"dns-default-mlc2p\" (UID: \"096c53b4-6072-4b7e-9d0c-f2ffb8e13694\") " pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:04:02.331713 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.331516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsc8s\" (UniqueName: \"kubernetes.io/projected/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-kube-api-access-bsc8s\") pod \"dns-default-mlc2p\" (UID: \"096c53b4-6072-4b7e-9d0c-f2ffb8e13694\") " pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:04:02.331713 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.331557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv644\" (UniqueName: \"kubernetes.io/projected/3b4372da-bcb1-4e89-8cd1-84642aa32972-kube-api-access-gv644\") pod \"ingress-canary-4hvw6\" (UID: \"3b4372da-bcb1-4e89-8cd1-84642aa32972\") " pod="openshift-ingress-canary/ingress-canary-4hvw6"
Sep 16 12:04:02.331837 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.331797    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-config-volume\") pod \"dns-default-mlc2p\" (UID: \"096c53b4-6072-4b7e-9d0c-f2ffb8e13694\") " pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:04:02.331951 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.331842    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b4372da-bcb1-4e89-8cd1-84642aa32972-cert\") pod \"ingress-canary-4hvw6\" (UID: \"3b4372da-bcb1-4e89-8cd1-84642aa32972\") " pod="openshift-ingress-canary/ingress-canary-4hvw6"
Sep 16 12:04:02.331951 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.331878    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls\") pod \"dns-default-mlc2p\" (UID: \"096c53b4-6072-4b7e-9d0c-f2ffb8e13694\") " pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:04:02.332058 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:02.331983    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:04:02.332058 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:02.332043    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls podName:096c53b4-6072-4b7e-9d0c-f2ffb8e13694 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:02.83202164 +0000 UTC m=+36.533092947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls") pod "dns-default-mlc2p" (UID: "096c53b4-6072-4b7e-9d0c-f2ffb8e13694") : secret "dns-default-metrics-tls" not found
Sep 16 12:04:02.332058 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.331846    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-tmp-dir\") pod \"dns-default-mlc2p\" (UID: \"096c53b4-6072-4b7e-9d0c-f2ffb8e13694\") " pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:04:02.332274 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.332246    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-config-volume\") pod \"dns-default-mlc2p\" (UID: \"096c53b4-6072-4b7e-9d0c-f2ffb8e13694\") " pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:04:02.337129 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.337103    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b4372da-bcb1-4e89-8cd1-84642aa32972-cert\") pod \"ingress-canary-4hvw6\" (UID: \"3b4372da-bcb1-4e89-8cd1-84642aa32972\") " pod="openshift-ingress-canary/ingress-canary-4hvw6"
Sep 16 12:04:02.340527 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.340502    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv644\" (UniqueName: \"kubernetes.io/projected/3b4372da-bcb1-4e89-8cd1-84642aa32972-kube-api-access-gv644\") pod \"ingress-canary-4hvw6\" (UID: \"3b4372da-bcb1-4e89-8cd1-84642aa32972\") " pod="openshift-ingress-canary/ingress-canary-4hvw6"
Sep 16 12:04:02.340714 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.340690    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsc8s\" (UniqueName: \"kubernetes.io/projected/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-kube-api-access-bsc8s\") pod \"dns-default-mlc2p\" (UID: \"096c53b4-6072-4b7e-9d0c-f2ffb8e13694\") " pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:04:02.435619 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.435584    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4hvw6"
Sep 16 12:04:02.835487 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:02.835453    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls\") pod \"dns-default-mlc2p\" (UID: \"096c53b4-6072-4b7e-9d0c-f2ffb8e13694\") " pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:04:02.835935 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:02.835599    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:04:02.835935 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:02.835684    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls podName:096c53b4-6072-4b7e-9d0c-f2ffb8e13694 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:03.835652441 +0000 UTC m=+37.536723744 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls") pod "dns-default-mlc2p" (UID: "096c53b4-6072-4b7e-9d0c-f2ffb8e13694") : secret "dns-default-metrics-tls" not found
Sep 16 12:04:03.263089 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:03.262920    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4hvw6"]
Sep 16 12:04:03.266112 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:04:03.266076    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4372da_bcb1_4e89_8cd1_84642aa32972.slice/crio-150a2aa82b55bfb363165543cde76e02b74a21392fb0f6850ecea5a733d2268e WatchSource:0}: Error finding container 150a2aa82b55bfb363165543cde76e02b74a21392fb0f6850ecea5a733d2268e: Status 404 returned error can't find the container with id 150a2aa82b55bfb363165543cde76e02b74a21392fb0f6850ecea5a733d2268e
Sep 16 12:04:03.774355 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:03.774285    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:04:03.774355 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:03.774337    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:04:03.774355 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:03.774338    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:04:03.777122 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:03.777092    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:04:03.777363 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:03.777137    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 12:04:03.777363 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:03.777185    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:04:03.778331 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:03.778297    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gvq92\""
Sep 16 12:04:03.778331 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:03.778317    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 12:04:03.778525 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:03.778355    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8kdnz\""
Sep 16 12:04:03.845646 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:03.845603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls\") pod \"dns-default-mlc2p\" (UID: \"096c53b4-6072-4b7e-9d0c-f2ffb8e13694\") " pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:04:03.846231 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:03.845783    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:04:03.846231 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:03.845865    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls podName:096c53b4-6072-4b7e-9d0c-f2ffb8e13694 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:05.845841862 +0000 UTC m=+39.546913172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls") pod "dns-default-mlc2p" (UID: "096c53b4-6072-4b7e-9d0c-f2ffb8e13694") : secret "dns-default-metrics-tls" not found
Sep 16 12:04:04.073475 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:04.073393    2579 generic.go:356] "Generic (PLEG): container finished" podID="6180fb10-c394-4bca-97af-1a28211ea33f" containerID="0ce4457395f383f5a8be7715dec4cc269f4d04d60be21a14efeb6c0979ebfbce" exitCode=0
Sep 16 12:04:04.073651 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:04.073487    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cwn4x" event={"ID":"6180fb10-c394-4bca-97af-1a28211ea33f","Type":"ContainerDied","Data":"0ce4457395f383f5a8be7715dec4cc269f4d04d60be21a14efeb6c0979ebfbce"}
Sep 16 12:04:04.074777 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:04.074741    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4hvw6" event={"ID":"3b4372da-bcb1-4e89-8cd1-84642aa32972","Type":"ContainerStarted","Data":"150a2aa82b55bfb363165543cde76e02b74a21392fb0f6850ecea5a733d2268e"}
Sep 16 12:04:05.081144 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:05.081116    2579 generic.go:356] "Generic (PLEG): container finished" podID="6180fb10-c394-4bca-97af-1a28211ea33f" containerID="5607555ea77bc596257ab4fc66e879c105a76bc23c8bcbdaec981296a3dc0ab0" exitCode=0
Sep 16 12:04:05.081681 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:05.081162    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cwn4x" event={"ID":"6180fb10-c394-4bca-97af-1a28211ea33f","Type":"ContainerDied","Data":"5607555ea77bc596257ab4fc66e879c105a76bc23c8bcbdaec981296a3dc0ab0"}
Sep 16 12:04:05.860623 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:05.860600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls\") pod \"dns-default-mlc2p\" (UID: \"096c53b4-6072-4b7e-9d0c-f2ffb8e13694\") " pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:04:05.860775 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:05.860756    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:04:05.860855 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:05.860828    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls podName:096c53b4-6072-4b7e-9d0c-f2ffb8e13694 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:09.860805815 +0000 UTC m=+43.561877125 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls") pod "dns-default-mlc2p" (UID: "096c53b4-6072-4b7e-9d0c-f2ffb8e13694") : secret "dns-default-metrics-tls" not found
Sep 16 12:04:06.087477 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:06.087437    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cwn4x" event={"ID":"6180fb10-c394-4bca-97af-1a28211ea33f","Type":"ContainerStarted","Data":"fc85c4c8712fa92f16f178b26c73367fa2af84ce1585a1f50d6012aff0f8cd0b"}
Sep 16 12:04:06.088951 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:06.088927    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4hvw6" event={"ID":"3b4372da-bcb1-4e89-8cd1-84642aa32972","Type":"ContainerStarted","Data":"5999f0fa8e60ab3d0dba05f0f059d9caa63575c78be4dd4898ccb576740e7af8"}
Sep 16 12:04:06.109224 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:06.109127    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cwn4x" podStartSLOduration=9.630294398 podStartE2EDuration="39.109110922s" totalImagesPullingTime="29.478816524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:04:06.107574278 +0000 UTC m=+39.808645606" watchObservedRunningTime="2026-09-16 12:04:06.109110922 +0000 UTC m=+39.810182283"
Sep 16 12:04:06.122482 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:06.122442    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4hvw6" podStartSLOduration=1.538201386 podStartE2EDuration="4.122427895s" totalImagesPullingTime="2.584226509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:04:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:04:06.121881696 +0000 UTC m=+39.822953023" watchObservedRunningTime="2026-09-16 12:04:06.122427895 +0000 UTC m=+39.823499221"
Sep 16 12:04:09.886691 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:09.886619    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls\") pod \"dns-default-mlc2p\" (UID: \"096c53b4-6072-4b7e-9d0c-f2ffb8e13694\") " pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:04:09.887061 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:09.886783    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:04:09.887061 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:09.886851    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls podName:096c53b4-6072-4b7e-9d0c-f2ffb8e13694 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:17.886833951 +0000 UTC m=+51.587905255 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls") pod "dns-default-mlc2p" (UID: "096c53b4-6072-4b7e-9d0c-f2ffb8e13694") : secret "dns-default-metrics-tls" not found
Sep 16 12:04:17.938046 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:17.938005    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls\") pod \"dns-default-mlc2p\" (UID: \"096c53b4-6072-4b7e-9d0c-f2ffb8e13694\") " pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:04:17.938519 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:17.938158    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:04:17.938519 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:17.938229    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls podName:096c53b4-6072-4b7e-9d0c-f2ffb8e13694 nodeName:}" failed. No retries permitted until 2026-09-16 12:04:33.938214555 +0000 UTC m=+67.639285858 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls") pod "dns-default-mlc2p" (UID: "096c53b4-6072-4b7e-9d0c-f2ffb8e13694") : secret "dns-default-metrics-tls" not found
Sep 16 12:04:21.559537 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.559504    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt"]
Sep 16 12:04:21.561706 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.561688    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt"
Sep 16 12:04:21.563613 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.563587    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86c756bfb-c7t5c"]
Sep 16 12:04:21.565110 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.565093    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86c756bfb-c7t5c"
Sep 16 12:04:21.566093 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.566070    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 12:04:21.566189 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.566078    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 12:04:21.566189 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.566102    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 12:04:21.566189 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.566116    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 12:04:21.567429 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.567411    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 12:04:21.567524 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.567467    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-5vvzl\""
Sep 16 12:04:21.573450 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.573424    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt"]
Sep 16 12:04:21.576860 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.576828    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86c756bfb-c7t5c"]
Sep 16 12:04:21.665165 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.665126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d227c519-3639-481a-bda3-280eb42dc754-tmp\") pod \"klusterlet-addon-workmgr-846f59685d-rk5qt\" (UID: \"d227c519-3639-481a-bda3-280eb42dc754\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt"
Sep 16 12:04:21.665165 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.665169    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw2zd\" (UniqueName: \"kubernetes.io/projected/28002685-3152-4c01-9543-56f847a1d212-kube-api-access-fw2zd\") pod \"managed-serviceaccount-addon-agent-86c756bfb-c7t5c\" (UID: \"28002685-3152-4c01-9543-56f847a1d212\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86c756bfb-c7t5c"
Sep 16 12:04:21.665367 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.665210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28002685-3152-4c01-9543-56f847a1d212-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-86c756bfb-c7t5c\" (UID: \"28002685-3152-4c01-9543-56f847a1d212\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86c756bfb-c7t5c"
Sep 16 12:04:21.665367 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.665228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jqgk\" (UniqueName: \"kubernetes.io/projected/d227c519-3639-481a-bda3-280eb42dc754-kube-api-access-6jqgk\") pod \"klusterlet-addon-workmgr-846f59685d-rk5qt\" (UID: \"d227c519-3639-481a-bda3-280eb42dc754\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt"
Sep 16 12:04:21.665367 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.665271    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d227c519-3639-481a-bda3-280eb42dc754-klusterlet-config\") pod \"klusterlet-addon-workmgr-846f59685d-rk5qt\" (UID: \"d227c519-3639-481a-bda3-280eb42dc754\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt"
Sep 16 12:04:21.766403 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.766375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d227c519-3639-481a-bda3-280eb42dc754-klusterlet-config\") pod \"klusterlet-addon-workmgr-846f59685d-rk5qt\" (UID: \"d227c519-3639-481a-bda3-280eb42dc754\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt"
Sep 16 12:04:21.766585 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.766425    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d227c519-3639-481a-bda3-280eb42dc754-tmp\") pod \"klusterlet-addon-workmgr-846f59685d-rk5qt\" (UID: \"d227c519-3639-481a-bda3-280eb42dc754\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt"
Sep 16 12:04:21.766585 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.766544    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw2zd\" (UniqueName: \"kubernetes.io/projected/28002685-3152-4c01-9543-56f847a1d212-kube-api-access-fw2zd\") pod \"managed-serviceaccount-addon-agent-86c756bfb-c7t5c\" (UID: \"28002685-3152-4c01-9543-56f847a1d212\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86c756bfb-c7t5c"
Sep 16 12:04:21.766722 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.766592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28002685-3152-4c01-9543-56f847a1d212-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-86c756bfb-c7t5c\" (UID: \"28002685-3152-4c01-9543-56f847a1d212\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86c756bfb-c7t5c"
Sep 16 12:04:21.766722 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.766621    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jqgk\" (UniqueName: \"kubernetes.io/projected/d227c519-3639-481a-bda3-280eb42dc754-kube-api-access-6jqgk\") pod \"klusterlet-addon-workmgr-846f59685d-rk5qt\" (UID: \"d227c519-3639-481a-bda3-280eb42dc754\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt"
Sep 16 12:04:21.766816 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.766747    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d227c519-3639-481a-bda3-280eb42dc754-tmp\") pod \"klusterlet-addon-workmgr-846f59685d-rk5qt\" (UID: \"d227c519-3639-481a-bda3-280eb42dc754\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt"
Sep 16 12:04:21.769629 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.769605    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d227c519-3639-481a-bda3-280eb42dc754-klusterlet-config\") pod \"klusterlet-addon-workmgr-846f59685d-rk5qt\" (UID: \"d227c519-3639-481a-bda3-280eb42dc754\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt"
Sep 16 12:04:21.769742 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.769684    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28002685-3152-4c01-9543-56f847a1d212-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-86c756bfb-c7t5c\" (UID: \"28002685-3152-4c01-9543-56f847a1d212\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86c756bfb-c7t5c"
Sep 16 12:04:21.775231 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.775199    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jqgk\" (UniqueName: \"kubernetes.io/projected/d227c519-3639-481a-bda3-280eb42dc754-kube-api-access-6jqgk\") pod \"klusterlet-addon-workmgr-846f59685d-rk5qt\" (UID: \"d227c519-3639-481a-bda3-280eb42dc754\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt"
Sep 16 12:04:21.775231 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.775224    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw2zd\" (UniqueName: \"kubernetes.io/projected/28002685-3152-4c01-9543-56f847a1d212-kube-api-access-fw2zd\") pod \"managed-serviceaccount-addon-agent-86c756bfb-c7t5c\" (UID: \"28002685-3152-4c01-9543-56f847a1d212\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86c756bfb-c7t5c"
Sep 16 12:04:21.872336 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.872301    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt"
Sep 16 12:04:21.888748 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:21.888725    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86c756bfb-c7t5c"
Sep 16 12:04:22.140136 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:22.140058    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt"]
Sep 16 12:04:22.143912 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:04:22.143885    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd227c519_3639_481a_bda3_280eb42dc754.slice/crio-7dc49ef2afa70903d14e4c5339e2ad56206d484aa9d944dfee66a82a7a1e0107 WatchSource:0}: Error finding container 7dc49ef2afa70903d14e4c5339e2ad56206d484aa9d944dfee66a82a7a1e0107: Status 404 returned error can't find the container with id 7dc49ef2afa70903d14e4c5339e2ad56206d484aa9d944dfee66a82a7a1e0107
Sep 16 12:04:22.161486 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:22.161464    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86c756bfb-c7t5c"]
Sep 16 12:04:22.164036 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:04:22.164009    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28002685_3152_4c01_9543_56f847a1d212.slice/crio-1bd1bd356ed602849173a096706aa809a0e547bd0427d726c1647a19c016ec55 WatchSource:0}: Error finding container 1bd1bd356ed602849173a096706aa809a0e547bd0427d726c1647a19c016ec55: Status 404 returned error can't find the container with id 1bd1bd356ed602849173a096706aa809a0e547bd0427d726c1647a19c016ec55
Sep 16 12:04:23.131819 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:23.131778    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt" event={"ID":"d227c519-3639-481a-bda3-280eb42dc754","Type":"ContainerStarted","Data":"7dc49ef2afa70903d14e4c5339e2ad56206d484aa9d944dfee66a82a7a1e0107"}
Sep 16 12:04:23.133435 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:23.133377    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86c756bfb-c7t5c" event={"ID":"28002685-3152-4c01-9543-56f847a1d212","Type":"ContainerStarted","Data":"1bd1bd356ed602849173a096706aa809a0e547bd0427d726c1647a19c016ec55"}
Sep 16 12:04:25.139776 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:25.139738    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86c756bfb-c7t5c" event={"ID":"28002685-3152-4c01-9543-56f847a1d212","Type":"ContainerStarted","Data":"8cfa8fa132823bd43bfe3b5010753105b2e9953108a6121f66fd4bc399f54b63"}
Sep 16 12:04:25.154232 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:25.154178    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86c756bfb-c7t5c" podStartSLOduration=1.631457095 podStartE2EDuration="4.154159244s" totalImagesPullingTime="2.522702149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:04:25.153849788 +0000 UTC m=+58.854921113" watchObservedRunningTime="2026-09-16 12:04:25.154159244 +0000 UTC m=+58.855230571"
Sep 16 12:04:27.062133 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:27.062101    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbbsg"
Sep 16 12:04:29.151471 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:29.151436    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt" event={"ID":"d227c519-3639-481a-bda3-280eb42dc754","Type":"ContainerStarted","Data":"91e97bca08073a099cedc7be349263067ee85d542a99d0030f6e317cdf23e3cd"}
Sep 16 12:04:29.151955 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:29.151703    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt"
Sep 16 12:04:29.153223 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:29.153204    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt"
Sep 16 12:04:29.167138 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:29.167097    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt" podStartSLOduration=2.092800695 podStartE2EDuration="8.167084723s" totalImagesPullingTime="6.074284028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:04:29.166090182 +0000 UTC m=+62.867161509" watchObservedRunningTime="2026-09-16 12:04:29.167084723 +0000 UTC m=+62.868156047"
Sep 16 12:04:32.441701 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:32.441641    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs\") pod \"network-metrics-daemon-7cn45\" (UID: \"5441211d-70c6-473a-ac90-417bd7aa57ec\") " pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:04:32.444376 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:32.444355    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 12:04:32.452482 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:32.452456    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 12:04:32.452547 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:32.452527    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs podName:5441211d-70c6-473a-ac90-417bd7aa57ec nodeName:}" failed. No retries permitted until 2026-09-16 12:05:36.452509669 +0000 UTC m=+130.153580973 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs") pod "network-metrics-daemon-7cn45" (UID: "5441211d-70c6-473a-ac90-417bd7aa57ec") : secret "metrics-daemon-secret" not found
Sep 16 12:04:32.542203 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:32.542155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fbbe06db-6359-4c0d-aa65-d07fa8245950-original-pull-secret\") pod \"global-pull-secret-syncer-qgx6n\" (UID: \"fbbe06db-6359-4c0d-aa65-d07fa8245950\") " pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:04:32.545100 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:32.545082    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 12:04:32.555331 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:32.555301    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fbbe06db-6359-4c0d-aa65-d07fa8245950-original-pull-secret\") pod \"global-pull-secret-syncer-qgx6n\" (UID: \"fbbe06db-6359-4c0d-aa65-d07fa8245950\") " pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:04:32.593583 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:32.593540    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qgx6n"
Sep 16 12:04:32.642530 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:32.642495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpfv9\" (UniqueName: \"kubernetes.io/projected/4fdacb7a-b65c-48f5-b4f3-6580109c3dbd-kube-api-access-bpfv9\") pod \"network-check-target-rbck7\" (UID: \"4fdacb7a-b65c-48f5-b4f3-6580109c3dbd\") " pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:04:32.645417 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:32.645397    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 12:04:32.655469 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:32.655442    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 12:04:32.667646 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:32.667620    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpfv9\" (UniqueName: \"kubernetes.io/projected/4fdacb7a-b65c-48f5-b4f3-6580109c3dbd-kube-api-access-bpfv9\") pod \"network-check-target-rbck7\" (UID: \"4fdacb7a-b65c-48f5-b4f3-6580109c3dbd\") " pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:04:32.881971 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:32.881937    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qgx6n"]
Sep 16 12:04:32.884915 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:04:32.884879    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbbe06db_6359_4c0d_aa65_d07fa8245950.slice/crio-0b52203cefbdbba9c89548a88a4601658af96139e0fe2bfc06dbbab55d00c70b WatchSource:0}: Error finding container 0b52203cefbdbba9c89548a88a4601658af96139e0fe2bfc06dbbab55d00c70b: Status 404 returned error can't find the container with id 0b52203cefbdbba9c89548a88a4601658af96139e0fe2bfc06dbbab55d00c70b
Sep 16 12:04:32.902552 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:32.902527    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gvq92\""
Sep 16 12:04:32.910449 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:32.910427    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:04:33.161860 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:33.161778    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qgx6n" event={"ID":"fbbe06db-6359-4c0d-aa65-d07fa8245950","Type":"ContainerStarted","Data":"0b52203cefbdbba9c89548a88a4601658af96139e0fe2bfc06dbbab55d00c70b"}
Sep 16 12:04:33.189764 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:33.189732    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rbck7"]
Sep 16 12:04:33.192524 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:04:33.192498    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fdacb7a_b65c_48f5_b4f3_6580109c3dbd.slice/crio-be17f246fd37aed9809537873c3a1860a96515f766de3a32541cbd987637017f WatchSource:0}: Error finding container be17f246fd37aed9809537873c3a1860a96515f766de3a32541cbd987637017f: Status 404 returned error can't find the container with id be17f246fd37aed9809537873c3a1860a96515f766de3a32541cbd987637017f
Sep 16 12:04:33.950882 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:33.950762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls\") pod \"dns-default-mlc2p\" (UID: \"096c53b4-6072-4b7e-9d0c-f2ffb8e13694\") " pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:04:33.951338 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:33.950928    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:04:33.951338 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:04:33.950982    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls podName:096c53b4-6072-4b7e-9d0c-f2ffb8e13694 nodeName:}" failed. No retries permitted until 2026-09-16 12:05:05.950968494 +0000 UTC m=+99.652039797 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls") pod "dns-default-mlc2p" (UID: "096c53b4-6072-4b7e-9d0c-f2ffb8e13694") : secret "dns-default-metrics-tls" not found
Sep 16 12:04:34.165958 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:34.165919    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rbck7" event={"ID":"4fdacb7a-b65c-48f5-b4f3-6580109c3dbd","Type":"ContainerStarted","Data":"be17f246fd37aed9809537873c3a1860a96515f766de3a32541cbd987637017f"}
Sep 16 12:04:38.178518 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:38.178478    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rbck7" event={"ID":"4fdacb7a-b65c-48f5-b4f3-6580109c3dbd","Type":"ContainerStarted","Data":"afb479440fb6a6788d6e4f08f0bd270da9bcd37d7134e66885259f3fd807ce3d"}
Sep 16 12:04:38.178995 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:38.178567    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:04:38.179866 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:38.179835    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qgx6n" event={"ID":"fbbe06db-6359-4c0d-aa65-d07fa8245950","Type":"ContainerStarted","Data":"4b7f730226220e70bdd8547d8e02604efd78dd98df642ce0198efe28768cf297"}
Sep 16 12:04:38.194968 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:38.194919    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rbck7" podStartSLOduration=66.840224681 podStartE2EDuration="1m11.194905068s" totalImagesPullingTime="4.354680387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:04:38.194756203 +0000 UTC m=+71.895827529" watchObservedRunningTime="2026-09-16 12:04:38.194905068 +0000 UTC m=+71.895976394"
Sep 16 12:04:38.209446 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:04:38.209405    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qgx6n" podStartSLOduration=66.543324497 podStartE2EDuration="1m11.209390843s" totalImagesPullingTime="4.666066346s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:04:38.208736551 +0000 UTC m=+71.909807877" watchObservedRunningTime="2026-09-16 12:04:38.209390843 +0000 UTC m=+71.910462169"
Sep 16 12:05:05.973559 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:05:05.973507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls\") pod \"dns-default-mlc2p\" (UID: \"096c53b4-6072-4b7e-9d0c-f2ffb8e13694\") " pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:05:05.974007 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:05:05.973633    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:05:05.974007 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:05:05.973723    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls podName:096c53b4-6072-4b7e-9d0c-f2ffb8e13694 nodeName:}" failed. No retries permitted until 2026-09-16 12:06:09.973708435 +0000 UTC m=+163.674779738 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls") pod "dns-default-mlc2p" (UID: "096c53b4-6072-4b7e-9d0c-f2ffb8e13694") : secret "dns-default-metrics-tls" not found
Sep 16 12:05:09.185078 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:05:09.185044    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rbck7"
Sep 16 12:05:36.483918 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:05:36.483875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs\") pod \"network-metrics-daemon-7cn45\" (UID: \"5441211d-70c6-473a-ac90-417bd7aa57ec\") " pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:05:36.484382 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:05:36.483989    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 12:05:36.484382 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:05:36.484042    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs podName:5441211d-70c6-473a-ac90-417bd7aa57ec nodeName:}" failed. No retries permitted until 2026-09-16 12:07:38.484027636 +0000 UTC m=+252.185098940 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs") pod "network-metrics-daemon-7cn45" (UID: "5441211d-70c6-473a-ac90-417bd7aa57ec") : secret "metrics-daemon-secret" not found
Sep 16 12:06:05.127758 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:06:05.127696    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mlc2p" podUID="096c53b4-6072-4b7e-9d0c-f2ffb8e13694"
Sep 16 12:06:05.389335 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:06:05.389257    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:06:06.786033 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:06:06.785983    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-7cn45" podUID="5441211d-70c6-473a-ac90-417bd7aa57ec"
Sep 16 12:06:10.013552 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:06:10.013502    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls\") pod \"dns-default-mlc2p\" (UID: \"096c53b4-6072-4b7e-9d0c-f2ffb8e13694\") " pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:06:10.013960 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:06:10.013639    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 12:06:10.013960 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:06:10.013727    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls podName:096c53b4-6072-4b7e-9d0c-f2ffb8e13694 nodeName:}" failed. No retries permitted until 2026-09-16 12:08:12.013710862 +0000 UTC m=+285.714782166 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls") pod "dns-default-mlc2p" (UID: "096c53b4-6072-4b7e-9d0c-f2ffb8e13694") : secret "dns-default-metrics-tls" not found
Sep 16 12:06:20.774709 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:06:20.774618    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:06:25.442934 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:06:25.442899    2579 generic.go:356] "Generic (PLEG): container finished" podID="28002685-3152-4c01-9543-56f847a1d212" containerID="8cfa8fa132823bd43bfe3b5010753105b2e9953108a6121f66fd4bc399f54b63" exitCode=255
Sep 16 12:06:25.442934 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:06:25.442934    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86c756bfb-c7t5c" event={"ID":"28002685-3152-4c01-9543-56f847a1d212","Type":"ContainerDied","Data":"8cfa8fa132823bd43bfe3b5010753105b2e9953108a6121f66fd4bc399f54b63"}
Sep 16 12:06:25.448423 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:06:25.448405    2579 scope.go:117] "RemoveContainer" containerID="8cfa8fa132823bd43bfe3b5010753105b2e9953108a6121f66fd4bc399f54b63"
Sep 16 12:06:26.448236 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:06:26.448198    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86c756bfb-c7t5c" event={"ID":"28002685-3152-4c01-9543-56f847a1d212","Type":"ContainerStarted","Data":"19eef8ebe6cbbbc66d72307352164a3c10dc3c80390f02f5d59f045e67444f8d"}
Sep 16 12:06:28.458098 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:06:28.458063    2579 generic.go:356] "Generic (PLEG): container finished" podID="d227c519-3639-481a-bda3-280eb42dc754" containerID="91e97bca08073a099cedc7be349263067ee85d542a99d0030f6e317cdf23e3cd" exitCode=1
Sep 16 12:06:28.458466 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:06:28.458112    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt" event={"ID":"d227c519-3639-481a-bda3-280eb42dc754","Type":"ContainerDied","Data":"91e97bca08073a099cedc7be349263067ee85d542a99d0030f6e317cdf23e3cd"}
Sep 16 12:06:28.458466 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:06:28.458403    2579 scope.go:117] "RemoveContainer" containerID="91e97bca08073a099cedc7be349263067ee85d542a99d0030f6e317cdf23e3cd"
Sep 16 12:06:29.152189 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:06:29.152143    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt"
Sep 16 12:06:29.462614 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:06:29.462535    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt" event={"ID":"d227c519-3639-481a-bda3-280eb42dc754","Type":"ContainerStarted","Data":"6d20a724787c386c94a908fa5278923783c0348c1b4e9fe8271b2970cc6460e0"}
Sep 16 12:06:29.462998 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:06:29.462726    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt"
Sep 16 12:06:29.463326 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:06:29.463309    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt"
Sep 16 12:07:38.496113 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:38.496058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs\") pod \"network-metrics-daemon-7cn45\" (UID: \"5441211d-70c6-473a-ac90-417bd7aa57ec\") " pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:07:38.496599 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:07:38.496204    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 12:07:38.496599 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:07:38.496273    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs podName:5441211d-70c6-473a-ac90-417bd7aa57ec nodeName:}" failed. No retries permitted until 2026-09-16 12:09:40.496259239 +0000 UTC m=+374.197330543 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs") pod "network-metrics-daemon-7cn45" (UID: "5441211d-70c6-473a-ac90-417bd7aa57ec") : secret "metrics-daemon-secret" not found
Sep 16 12:07:52.814779 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.814684    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-8f59b"]
Sep 16 12:07:52.816537 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.816516    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5b5wz"]
Sep 16 12:07:52.816678 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.816630    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-8f59b"
Sep 16 12:07:52.818110 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.818092    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5b5wz"
Sep 16 12:07:52.819126 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.819099    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 12:07:52.820560 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.820540    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 12:07:52.820917 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.820900    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 12:07:52.821894 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.821876    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-fhhxw\""
Sep 16 12:07:52.822128 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.821946    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:07:52.822128 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.821962    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-zkvrr\""
Sep 16 12:07:52.822128 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.822033    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:07:52.823687 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.823651    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 16 12:07:52.828074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.828049    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-8f59b"]
Sep 16 12:07:52.833054 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.833029    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5b5wz"]
Sep 16 12:07:52.883749 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.883713    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pw7p9"]
Sep 16 12:07:52.885648 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.885629    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-xqnvz"]
Sep 16 12:07:52.885760 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.885741    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pw7p9"
Sep 16 12:07:52.887414 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.887394    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xqnvz"
Sep 16 12:07:52.888886 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.888866    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:07:52.889000 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.888934    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 12:07:52.889546 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.889525    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 12:07:52.889775 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.889759    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 12:07:52.889866 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.889762    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-xvq6j\""
Sep 16 12:07:52.890073 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.890051    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:07:52.890176 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.890095    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 12:07:52.890176 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.890057    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 12:07:52.890176 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.890059    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-l6mk6\""
Sep 16 12:07:52.892225 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.892180    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2dca0f8-99b0-4189-b1c0-329c54fdbc24-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-8f59b\" (UID: \"f2dca0f8-99b0-4189-b1c0-329c54fdbc24\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-8f59b"
Sep 16 12:07:52.892354 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.892259    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtqcg\" (UniqueName: \"kubernetes.io/projected/da760c1f-86c9-4430-a440-34b5dcf6a5c4-kube-api-access-dtqcg\") pod \"volume-data-source-validator-86695b7cc7-5b5wz\" (UID: \"da760c1f-86c9-4430-a440-34b5dcf6a5c4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5b5wz"
Sep 16 12:07:52.892354 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.892301    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f2dca0f8-99b0-4189-b1c0-329c54fdbc24-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-8f59b\" (UID: \"f2dca0f8-99b0-4189-b1c0-329c54fdbc24\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-8f59b"
Sep 16 12:07:52.892354 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.892329    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnrvz\" (UniqueName: \"kubernetes.io/projected/f2dca0f8-99b0-4189-b1c0-329c54fdbc24-kube-api-access-gnrvz\") pod \"kube-storage-version-migrator-operator-6c8f84f897-8f59b\" (UID: \"f2dca0f8-99b0-4189-b1c0-329c54fdbc24\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-8f59b"
Sep 16 12:07:52.898157 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.898124    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-xqnvz"]
Sep 16 12:07:52.899935 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.899904    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pw7p9"]
Sep 16 12:07:52.993110 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.993073    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gtxq\" (UniqueName: \"kubernetes.io/projected/e71d01c7-f1b2-44e0-b69d-94cf24847b6d-kube-api-access-5gtxq\") pod \"cluster-samples-operator-86499c89c4-pw7p9\" (UID: \"e71d01c7-f1b2-44e0-b69d-94cf24847b6d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pw7p9"
Sep 16 12:07:52.993110 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.993110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4j67\" (UniqueName: \"kubernetes.io/projected/d7c6dfde-e151-4ca0-86bf-3cd8b766ee53-kube-api-access-c4j67\") pod \"service-ca-operator-cf7c776b7-xqnvz\" (UID: \"d7c6dfde-e151-4ca0-86bf-3cd8b766ee53\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xqnvz"
Sep 16 12:07:52.993265 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.993129    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e71d01c7-f1b2-44e0-b69d-94cf24847b6d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-pw7p9\" (UID: \"e71d01c7-f1b2-44e0-b69d-94cf24847b6d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pw7p9"
Sep 16 12:07:52.993265 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.993156    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d7c6dfde-e151-4ca0-86bf-3cd8b766ee53-serving-cert\") pod \"service-ca-operator-cf7c776b7-xqnvz\" (UID: \"d7c6dfde-e151-4ca0-86bf-3cd8b766ee53\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xqnvz"
Sep 16 12:07:52.993265 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.993188    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d7c6dfde-e151-4ca0-86bf-3cd8b766ee53-config\") pod \"service-ca-operator-cf7c776b7-xqnvz\" (UID: \"d7c6dfde-e151-4ca0-86bf-3cd8b766ee53\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xqnvz"
Sep 16 12:07:52.993265 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.993212    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2dca0f8-99b0-4189-b1c0-329c54fdbc24-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-8f59b\" (UID: \"f2dca0f8-99b0-4189-b1c0-329c54fdbc24\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-8f59b"
Sep 16 12:07:52.993440 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.993257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtqcg\" (UniqueName: \"kubernetes.io/projected/da760c1f-86c9-4430-a440-34b5dcf6a5c4-kube-api-access-dtqcg\") pod \"volume-data-source-validator-86695b7cc7-5b5wz\" (UID: \"da760c1f-86c9-4430-a440-34b5dcf6a5c4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5b5wz"
Sep 16 12:07:52.993440 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.993288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f2dca0f8-99b0-4189-b1c0-329c54fdbc24-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-8f59b\" (UID: \"f2dca0f8-99b0-4189-b1c0-329c54fdbc24\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-8f59b"
Sep 16 12:07:52.993440 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.993307    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnrvz\" (UniqueName: \"kubernetes.io/projected/f2dca0f8-99b0-4189-b1c0-329c54fdbc24-kube-api-access-gnrvz\") pod \"kube-storage-version-migrator-operator-6c8f84f897-8f59b\" (UID: \"f2dca0f8-99b0-4189-b1c0-329c54fdbc24\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-8f59b"
Sep 16 12:07:52.993900 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.993880    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f2dca0f8-99b0-4189-b1c0-329c54fdbc24-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-8f59b\" (UID: \"f2dca0f8-99b0-4189-b1c0-329c54fdbc24\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-8f59b"
Sep 16 12:07:52.995422 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:52.995396    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2dca0f8-99b0-4189-b1c0-329c54fdbc24-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-8f59b\" (UID: \"f2dca0f8-99b0-4189-b1c0-329c54fdbc24\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-8f59b"
Sep 16 12:07:53.002958 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:53.002927    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnrvz\" (UniqueName: \"kubernetes.io/projected/f2dca0f8-99b0-4189-b1c0-329c54fdbc24-kube-api-access-gnrvz\") pod \"kube-storage-version-migrator-operator-6c8f84f897-8f59b\" (UID: \"f2dca0f8-99b0-4189-b1c0-329c54fdbc24\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-8f59b"
Sep 16 12:07:53.002958 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:53.002939    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtqcg\" (UniqueName: \"kubernetes.io/projected/da760c1f-86c9-4430-a440-34b5dcf6a5c4-kube-api-access-dtqcg\") pod \"volume-data-source-validator-86695b7cc7-5b5wz\" (UID: \"da760c1f-86c9-4430-a440-34b5dcf6a5c4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5b5wz"
Sep 16 12:07:53.093909 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:53.093879    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gtxq\" (UniqueName: \"kubernetes.io/projected/e71d01c7-f1b2-44e0-b69d-94cf24847b6d-kube-api-access-5gtxq\") pod \"cluster-samples-operator-86499c89c4-pw7p9\" (UID: \"e71d01c7-f1b2-44e0-b69d-94cf24847b6d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pw7p9"
Sep 16 12:07:53.094058 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:53.093912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4j67\" (UniqueName: \"kubernetes.io/projected/d7c6dfde-e151-4ca0-86bf-3cd8b766ee53-kube-api-access-c4j67\") pod \"service-ca-operator-cf7c776b7-xqnvz\" (UID: \"d7c6dfde-e151-4ca0-86bf-3cd8b766ee53\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xqnvz"
Sep 16 12:07:53.094058 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:53.093929    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e71d01c7-f1b2-44e0-b69d-94cf24847b6d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-pw7p9\" (UID: \"e71d01c7-f1b2-44e0-b69d-94cf24847b6d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pw7p9"
Sep 16 12:07:53.094058 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:53.093953    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d7c6dfde-e151-4ca0-86bf-3cd8b766ee53-serving-cert\") pod \"service-ca-operator-cf7c776b7-xqnvz\" (UID: \"d7c6dfde-e151-4ca0-86bf-3cd8b766ee53\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xqnvz"
Sep 16 12:07:53.094058 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:53.093994    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d7c6dfde-e151-4ca0-86bf-3cd8b766ee53-config\") pod \"service-ca-operator-cf7c776b7-xqnvz\" (UID: \"d7c6dfde-e151-4ca0-86bf-3cd8b766ee53\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xqnvz"
Sep 16 12:07:53.094223 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:07:53.094053    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:07:53.094223 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:07:53.094127    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e71d01c7-f1b2-44e0-b69d-94cf24847b6d-samples-operator-tls podName:e71d01c7-f1b2-44e0-b69d-94cf24847b6d nodeName:}" failed. No retries permitted until 2026-09-16 12:07:53.594106782 +0000 UTC m=+267.295178099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e71d01c7-f1b2-44e0-b69d-94cf24847b6d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-pw7p9" (UID: "e71d01c7-f1b2-44e0-b69d-94cf24847b6d") : secret "samples-operator-tls" not found
Sep 16 12:07:53.094578 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:53.094562    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d7c6dfde-e151-4ca0-86bf-3cd8b766ee53-config\") pod \"service-ca-operator-cf7c776b7-xqnvz\" (UID: \"d7c6dfde-e151-4ca0-86bf-3cd8b766ee53\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xqnvz"
Sep 16 12:07:53.096093 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:53.096074    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d7c6dfde-e151-4ca0-86bf-3cd8b766ee53-serving-cert\") pod \"service-ca-operator-cf7c776b7-xqnvz\" (UID: \"d7c6dfde-e151-4ca0-86bf-3cd8b766ee53\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xqnvz"
Sep 16 12:07:53.101886 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:53.101860    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4j67\" (UniqueName: \"kubernetes.io/projected/d7c6dfde-e151-4ca0-86bf-3cd8b766ee53-kube-api-access-c4j67\") pod \"service-ca-operator-cf7c776b7-xqnvz\" (UID: \"d7c6dfde-e151-4ca0-86bf-3cd8b766ee53\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xqnvz"
Sep 16 12:07:53.102011 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:53.101920    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gtxq\" (UniqueName: \"kubernetes.io/projected/e71d01c7-f1b2-44e0-b69d-94cf24847b6d-kube-api-access-5gtxq\") pod \"cluster-samples-operator-86499c89c4-pw7p9\" (UID: \"e71d01c7-f1b2-44e0-b69d-94cf24847b6d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pw7p9"
Sep 16 12:07:53.127761 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:53.127738    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-8f59b"
Sep 16 12:07:53.133168 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:53.133152    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5b5wz"
Sep 16 12:07:53.203820 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:53.203797    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xqnvz"
Sep 16 12:07:53.399417 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:53.399332    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-8f59b"]
Sep 16 12:07:53.402029 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:07:53.401991    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2dca0f8_99b0_4189_b1c0_329c54fdbc24.slice/crio-1831ca87cfa506cb1a47dc269080544cb5aeb285cebc0836fc78823f4afafd3c WatchSource:0}: Error finding container 1831ca87cfa506cb1a47dc269080544cb5aeb285cebc0836fc78823f4afafd3c: Status 404 returned error can't find the container with id 1831ca87cfa506cb1a47dc269080544cb5aeb285cebc0836fc78823f4afafd3c
Sep 16 12:07:53.403518 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:53.403493    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5b5wz"]
Sep 16 12:07:53.406780 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:07:53.406754    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda760c1f_86c9_4430_a440_34b5dcf6a5c4.slice/crio-dc89bfcc373ba09970ea543c15e97469b992fe55c4a7633f354cd60ef245f58e WatchSource:0}: Error finding container dc89bfcc373ba09970ea543c15e97469b992fe55c4a7633f354cd60ef245f58e: Status 404 returned error can't find the container with id dc89bfcc373ba09970ea543c15e97469b992fe55c4a7633f354cd60ef245f58e
Sep 16 12:07:53.473403 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:53.473371    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-xqnvz"]
Sep 16 12:07:53.477145 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:07:53.477112    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c6dfde_e151_4ca0_86bf_3cd8b766ee53.slice/crio-cb6a3af3f6281657344684449b3f3b3757f957b3b04ae7bb0b9ce1ce997df484 WatchSource:0}: Error finding container cb6a3af3f6281657344684449b3f3b3757f957b3b04ae7bb0b9ce1ce997df484: Status 404 returned error can't find the container with id cb6a3af3f6281657344684449b3f3b3757f957b3b04ae7bb0b9ce1ce997df484
Sep 16 12:07:53.597594 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:53.597554    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e71d01c7-f1b2-44e0-b69d-94cf24847b6d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-pw7p9\" (UID: \"e71d01c7-f1b2-44e0-b69d-94cf24847b6d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pw7p9"
Sep 16 12:07:53.597806 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:07:53.597744    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:07:53.597873 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:07:53.597820    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e71d01c7-f1b2-44e0-b69d-94cf24847b6d-samples-operator-tls podName:e71d01c7-f1b2-44e0-b69d-94cf24847b6d nodeName:}" failed. No retries permitted until 2026-09-16 12:07:54.597798446 +0000 UTC m=+268.298869751 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e71d01c7-f1b2-44e0-b69d-94cf24847b6d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-pw7p9" (UID: "e71d01c7-f1b2-44e0-b69d-94cf24847b6d") : secret "samples-operator-tls" not found
Sep 16 12:07:53.698149 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:53.698053    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xqnvz" event={"ID":"d7c6dfde-e151-4ca0-86bf-3cd8b766ee53","Type":"ContainerStarted","Data":"cb6a3af3f6281657344684449b3f3b3757f957b3b04ae7bb0b9ce1ce997df484"}
Sep 16 12:07:53.699176 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:53.699151    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-8f59b" event={"ID":"f2dca0f8-99b0-4189-b1c0-329c54fdbc24","Type":"ContainerStarted","Data":"1831ca87cfa506cb1a47dc269080544cb5aeb285cebc0836fc78823f4afafd3c"}
Sep 16 12:07:53.700217 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:53.700196    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5b5wz" event={"ID":"da760c1f-86c9-4430-a440-34b5dcf6a5c4","Type":"ContainerStarted","Data":"dc89bfcc373ba09970ea543c15e97469b992fe55c4a7633f354cd60ef245f58e"}
Sep 16 12:07:54.606301 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:54.606238    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e71d01c7-f1b2-44e0-b69d-94cf24847b6d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-pw7p9\" (UID: \"e71d01c7-f1b2-44e0-b69d-94cf24847b6d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pw7p9"
Sep 16 12:07:54.606766 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:07:54.606412    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:07:54.606766 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:07:54.606492    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e71d01c7-f1b2-44e0-b69d-94cf24847b6d-samples-operator-tls podName:e71d01c7-f1b2-44e0-b69d-94cf24847b6d nodeName:}" failed. No retries permitted until 2026-09-16 12:07:56.606470501 +0000 UTC m=+270.307541821 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e71d01c7-f1b2-44e0-b69d-94cf24847b6d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-pw7p9" (UID: "e71d01c7-f1b2-44e0-b69d-94cf24847b6d") : secret "samples-operator-tls" not found
Sep 16 12:07:55.710610 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:55.710571    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5b5wz" event={"ID":"da760c1f-86c9-4430-a440-34b5dcf6a5c4","Type":"ContainerStarted","Data":"b8ab9f4f7e421bede47fd3690859a357c61c0132252490bb9a784cef7a07a893"}
Sep 16 12:07:55.726148 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:55.726099    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-5b5wz" podStartSLOduration=2.408748848 podStartE2EDuration="3.72608034s" totalImagesPullingTime="1.317331492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:07:55.72474073 +0000 UTC m=+269.425812056" watchObservedRunningTime="2026-09-16 12:07:55.72608034 +0000 UTC m=+269.427151664"
Sep 16 12:07:56.625674 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:56.625631    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e71d01c7-f1b2-44e0-b69d-94cf24847b6d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-pw7p9\" (UID: \"e71d01c7-f1b2-44e0-b69d-94cf24847b6d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pw7p9"
Sep 16 12:07:56.625849 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:07:56.625805    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:07:56.625899 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:07:56.625887    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e71d01c7-f1b2-44e0-b69d-94cf24847b6d-samples-operator-tls podName:e71d01c7-f1b2-44e0-b69d-94cf24847b6d nodeName:}" failed. No retries permitted until 2026-09-16 12:08:00.625866283 +0000 UTC m=+274.326937594 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e71d01c7-f1b2-44e0-b69d-94cf24847b6d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-pw7p9" (UID: "e71d01c7-f1b2-44e0-b69d-94cf24847b6d") : secret "samples-operator-tls" not found
Sep 16 12:07:56.716593 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:56.716552    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xqnvz" event={"ID":"d7c6dfde-e151-4ca0-86bf-3cd8b766ee53","Type":"ContainerStarted","Data":"0b2d628e429583115dc2a7e3fd4747929532fa587e4284460a9e5247d25be259"}
Sep 16 12:07:56.718412 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:56.718385    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-8f59b" event={"ID":"f2dca0f8-99b0-4189-b1c0-329c54fdbc24","Type":"ContainerStarted","Data":"f6069bf683fa51d5f49b2e21c651675da928d796dd792c132f93ecc1b2f34b8b"}
Sep 16 12:07:56.731821 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:56.731770    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xqnvz" podStartSLOduration=2.105804447 podStartE2EDuration="4.73175462s" totalImagesPullingTime="2.625950173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:07:56.730936048 +0000 UTC m=+270.432007375" watchObservedRunningTime="2026-09-16 12:07:56.73175462 +0000 UTC m=+270.432825945"
Sep 16 12:07:56.744978 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:56.744908    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-8f59b" podStartSLOduration=2.047423109 podStartE2EDuration="4.744890476s" totalImagesPullingTime="2.697467367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:07:56.744644824 +0000 UTC m=+270.445716156" watchObservedRunningTime="2026-09-16 12:07:56.744890476 +0000 UTC m=+270.445961802"
Sep 16 12:07:56.868675 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:56.868613    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sx7z6"]
Sep 16 12:07:56.873592 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:56.873562    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:07:56.876049 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:56.875977    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7hqfk\""
Sep 16 12:07:56.876049 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:56.875993    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 12:07:56.876218 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:56.876050    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 12:07:56.876218 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:56.876086    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 12:07:56.876440 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:56.876420    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 12:07:56.880230 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:56.880210    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sx7z6"]
Sep 16 12:07:56.929211 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:56.929167    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85173723-783e-41a8-8b46-0459ca49e9e1-crio-socket\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:07:56.929211 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:56.929210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85173723-783e-41a8-8b46-0459ca49e9e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:07:56.929469 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:56.929239    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85173723-783e-41a8-8b46-0459ca49e9e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:07:56.929469 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:56.929316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85173723-783e-41a8-8b46-0459ca49e9e1-data-volume\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:07:56.929469 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:56.929431    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj52m\" (UniqueName: \"kubernetes.io/projected/85173723-783e-41a8-8b46-0459ca49e9e1-kube-api-access-fj52m\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:07:57.030579 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:57.030538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj52m\" (UniqueName: \"kubernetes.io/projected/85173723-783e-41a8-8b46-0459ca49e9e1-kube-api-access-fj52m\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:07:57.030776 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:57.030604    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85173723-783e-41a8-8b46-0459ca49e9e1-crio-socket\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:07:57.030776 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:57.030629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85173723-783e-41a8-8b46-0459ca49e9e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:07:57.030776 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:57.030649    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85173723-783e-41a8-8b46-0459ca49e9e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:07:57.030776 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:57.030718    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85173723-783e-41a8-8b46-0459ca49e9e1-data-volume\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:07:57.030776 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:57.030736    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85173723-783e-41a8-8b46-0459ca49e9e1-crio-socket\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:07:57.030776 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:07:57.030740    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:07:57.031128 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:07:57.030802    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:07:57.031128 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:07:57.030866    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85173723-783e-41a8-8b46-0459ca49e9e1-kube-rbac-proxy-cm podName:85173723-783e-41a8-8b46-0459ca49e9e1 nodeName:}" failed. No retries permitted until 2026-09-16 12:07:57.530844618 +0000 UTC m=+271.231915936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85173723-783e-41a8-8b46-0459ca49e9e1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sx7z6" (UID: "85173723-783e-41a8-8b46-0459ca49e9e1") : configmap "kube-rbac-proxy" not found
Sep 16 12:07:57.031128 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:07:57.030887    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85173723-783e-41a8-8b46-0459ca49e9e1-insights-runtime-extractor-tls podName:85173723-783e-41a8-8b46-0459ca49e9e1 nodeName:}" failed. No retries permitted until 2026-09-16 12:07:57.530875989 +0000 UTC m=+271.231947300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85173723-783e-41a8-8b46-0459ca49e9e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sx7z6" (UID: "85173723-783e-41a8-8b46-0459ca49e9e1") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:07:57.031128 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:57.031097    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85173723-783e-41a8-8b46-0459ca49e9e1-data-volume\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:07:57.046529 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:57.046502    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj52m\" (UniqueName: \"kubernetes.io/projected/85173723-783e-41a8-8b46-0459ca49e9e1-kube-api-access-fj52m\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:07:57.095055 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:57.095021    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-6m4db"]
Sep 16 12:07:57.098143 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:57.098125    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-6m4db"
Sep 16 12:07:57.100898 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:57.100874    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6l4rn\""
Sep 16 12:07:57.100898 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:57.100888    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 12:07:57.101094 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:57.100933    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 12:07:57.106872 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:57.106846    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-6m4db"]
Sep 16 12:07:57.232582 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:57.232494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hskrz\" (UniqueName: \"kubernetes.io/projected/49ccefde-2d94-4218-adef-2f8d3a268331-kube-api-access-hskrz\") pod \"migrator-8b4d697c5-6m4db\" (UID: \"49ccefde-2d94-4218-adef-2f8d3a268331\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-6m4db"
Sep 16 12:07:57.333279 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:57.333226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hskrz\" (UniqueName: \"kubernetes.io/projected/49ccefde-2d94-4218-adef-2f8d3a268331-kube-api-access-hskrz\") pod \"migrator-8b4d697c5-6m4db\" (UID: \"49ccefde-2d94-4218-adef-2f8d3a268331\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-6m4db"
Sep 16 12:07:57.340585 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:57.340556    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hskrz\" (UniqueName: \"kubernetes.io/projected/49ccefde-2d94-4218-adef-2f8d3a268331-kube-api-access-hskrz\") pod \"migrator-8b4d697c5-6m4db\" (UID: \"49ccefde-2d94-4218-adef-2f8d3a268331\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-6m4db"
Sep 16 12:07:57.407809 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:57.407774    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-6m4db"
Sep 16 12:07:57.534414 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:57.534314    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85173723-783e-41a8-8b46-0459ca49e9e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:07:57.534414 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:57.534366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85173723-783e-41a8-8b46-0459ca49e9e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:07:57.534636 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:07:57.534432    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:07:57.534636 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:07:57.534499    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85173723-783e-41a8-8b46-0459ca49e9e1-kube-rbac-proxy-cm podName:85173723-783e-41a8-8b46-0459ca49e9e1 nodeName:}" failed. No retries permitted until 2026-09-16 12:07:58.534484172 +0000 UTC m=+272.235555480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85173723-783e-41a8-8b46-0459ca49e9e1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sx7z6" (UID: "85173723-783e-41a8-8b46-0459ca49e9e1") : configmap "kube-rbac-proxy" not found
Sep 16 12:07:57.534636 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:07:57.534544    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:07:57.534636 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:07:57.534609    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85173723-783e-41a8-8b46-0459ca49e9e1-insights-runtime-extractor-tls podName:85173723-783e-41a8-8b46-0459ca49e9e1 nodeName:}" failed. No retries permitted until 2026-09-16 12:07:58.534590872 +0000 UTC m=+272.235662183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85173723-783e-41a8-8b46-0459ca49e9e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sx7z6" (UID: "85173723-783e-41a8-8b46-0459ca49e9e1") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:07:57.674433 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:57.674349    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-6m4db"]
Sep 16 12:07:57.679864 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:07:57.679839    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ccefde_2d94_4218_adef_2f8d3a268331.slice/crio-722bc31d5a2c326ecf9598c5898f020f005c0f1ec68fe2f1f4ed6d9c6649284a WatchSource:0}: Error finding container 722bc31d5a2c326ecf9598c5898f020f005c0f1ec68fe2f1f4ed6d9c6649284a: Status 404 returned error can't find the container with id 722bc31d5a2c326ecf9598c5898f020f005c0f1ec68fe2f1f4ed6d9c6649284a
Sep 16 12:07:57.723959 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:57.723927    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-6m4db" event={"ID":"49ccefde-2d94-4218-adef-2f8d3a268331","Type":"ContainerStarted","Data":"722bc31d5a2c326ecf9598c5898f020f005c0f1ec68fe2f1f4ed6d9c6649284a"}
Sep 16 12:07:58.541811 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:58.541773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85173723-783e-41a8-8b46-0459ca49e9e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:07:58.542074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:58.541824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85173723-783e-41a8-8b46-0459ca49e9e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:07:58.542074 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:07:58.541913    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:07:58.542074 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:07:58.541991    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85173723-783e-41a8-8b46-0459ca49e9e1-kube-rbac-proxy-cm podName:85173723-783e-41a8-8b46-0459ca49e9e1 nodeName:}" failed. No retries permitted until 2026-09-16 12:08:00.541969091 +0000 UTC m=+274.243040412 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85173723-783e-41a8-8b46-0459ca49e9e1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sx7z6" (UID: "85173723-783e-41a8-8b46-0459ca49e9e1") : configmap "kube-rbac-proxy" not found
Sep 16 12:07:58.542074 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:07:58.542048    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:07:58.542361 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:07:58.542100    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85173723-783e-41a8-8b46-0459ca49e9e1-insights-runtime-extractor-tls podName:85173723-783e-41a8-8b46-0459ca49e9e1 nodeName:}" failed. No retries permitted until 2026-09-16 12:08:00.542084618 +0000 UTC m=+274.243155923 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85173723-783e-41a8-8b46-0459ca49e9e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sx7z6" (UID: "85173723-783e-41a8-8b46-0459ca49e9e1") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:07:59.027503 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:59.027484    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q4hbf_b38a0789-177d-497c-8d7f-7d7177f5015e/dns-node-resolver/0.log"
Sep 16 12:07:59.627643 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:59.627615    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ldcrr_a6a91efe-a2f3-4829-9777-1ea089f08030/node-ca/0.log"
Sep 16 12:07:59.731983 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:59.731945    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-6m4db" event={"ID":"49ccefde-2d94-4218-adef-2f8d3a268331","Type":"ContainerStarted","Data":"de6cd586fe4b877a0b268a241bf6aac3b8c7c96aa116e79ddf3ac62f66a29048"}
Sep 16 12:07:59.732168 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:59.731997    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-6m4db" event={"ID":"49ccefde-2d94-4218-adef-2f8d3a268331","Type":"ContainerStarted","Data":"c2c4f813382e56892fcbfaf49f33d28f267dce4bb698e901febcd950f9ef0f39"}
Sep 16 12:07:59.746327 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:07:59.746280    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-6m4db" podStartSLOduration=1.519382523 podStartE2EDuration="2.746265417s" totalImagesPullingTime="1.226882894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:07:59.745865596 +0000 UTC m=+273.446936935" watchObservedRunningTime="2026-09-16 12:07:59.746265417 +0000 UTC m=+273.447336744"
Sep 16 12:08:00.435674 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:00.435632    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4hvw6_3b4372da-bcb1-4e89-8cd1-84642aa32972/serve-healthcheck-canary/0.log"
Sep 16 12:08:00.558291 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:00.558255    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85173723-783e-41a8-8b46-0459ca49e9e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:08:00.558291 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:00.558293    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85173723-783e-41a8-8b46-0459ca49e9e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:08:00.558502 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:08:00.558386    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:08:00.558502 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:08:00.558453    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85173723-783e-41a8-8b46-0459ca49e9e1-kube-rbac-proxy-cm podName:85173723-783e-41a8-8b46-0459ca49e9e1 nodeName:}" failed. No retries permitted until 2026-09-16 12:08:04.558433928 +0000 UTC m=+278.259505245 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85173723-783e-41a8-8b46-0459ca49e9e1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sx7z6" (UID: "85173723-783e-41a8-8b46-0459ca49e9e1") : configmap "kube-rbac-proxy" not found
Sep 16 12:08:00.558502 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:08:00.558461    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:08:00.558502 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:08:00.558493    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85173723-783e-41a8-8b46-0459ca49e9e1-insights-runtime-extractor-tls podName:85173723-783e-41a8-8b46-0459ca49e9e1 nodeName:}" failed. No retries permitted until 2026-09-16 12:08:04.558483802 +0000 UTC m=+278.259555109 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85173723-783e-41a8-8b46-0459ca49e9e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sx7z6" (UID: "85173723-783e-41a8-8b46-0459ca49e9e1") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:08:00.659549 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:00.659511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e71d01c7-f1b2-44e0-b69d-94cf24847b6d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-pw7p9\" (UID: \"e71d01c7-f1b2-44e0-b69d-94cf24847b6d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pw7p9"
Sep 16 12:08:00.659731 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:08:00.659650    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 12:08:00.659785 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:08:00.659743    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e71d01c7-f1b2-44e0-b69d-94cf24847b6d-samples-operator-tls podName:e71d01c7-f1b2-44e0-b69d-94cf24847b6d nodeName:}" failed. No retries permitted until 2026-09-16 12:08:08.659728073 +0000 UTC m=+282.360799381 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e71d01c7-f1b2-44e0-b69d-94cf24847b6d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-pw7p9" (UID: "e71d01c7-f1b2-44e0-b69d-94cf24847b6d") : secret "samples-operator-tls" not found
Sep 16 12:08:04.593159 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:04.593124    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85173723-783e-41a8-8b46-0459ca49e9e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:08:04.593159 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:04.593166    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85173723-783e-41a8-8b46-0459ca49e9e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:08:04.593608 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:08:04.593252    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 12:08:04.593608 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:08:04.593313    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85173723-783e-41a8-8b46-0459ca49e9e1-kube-rbac-proxy-cm podName:85173723-783e-41a8-8b46-0459ca49e9e1 nodeName:}" failed. No retries permitted until 2026-09-16 12:08:12.593299558 +0000 UTC m=+286.294370868 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85173723-783e-41a8-8b46-0459ca49e9e1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sx7z6" (UID: "85173723-783e-41a8-8b46-0459ca49e9e1") : configmap "kube-rbac-proxy" not found
Sep 16 12:08:04.593608 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:08:04.593348    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:08:04.593608 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:08:04.593411    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85173723-783e-41a8-8b46-0459ca49e9e1-insights-runtime-extractor-tls podName:85173723-783e-41a8-8b46-0459ca49e9e1 nodeName:}" failed. No retries permitted until 2026-09-16 12:08:12.593398056 +0000 UTC m=+286.294469360 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85173723-783e-41a8-8b46-0459ca49e9e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sx7z6" (UID: "85173723-783e-41a8-8b46-0459ca49e9e1") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:08:08.390788 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:08:08.390730    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mlc2p" podUID="096c53b4-6072-4b7e-9d0c-f2ffb8e13694"
Sep 16 12:08:08.724762 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:08.724677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e71d01c7-f1b2-44e0-b69d-94cf24847b6d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-pw7p9\" (UID: \"e71d01c7-f1b2-44e0-b69d-94cf24847b6d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pw7p9"
Sep 16 12:08:08.727024 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:08.727002    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e71d01c7-f1b2-44e0-b69d-94cf24847b6d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-pw7p9\" (UID: \"e71d01c7-f1b2-44e0-b69d-94cf24847b6d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pw7p9"
Sep 16 12:08:08.761765 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:08.761739    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:08:08.797196 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:08.797164    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pw7p9"
Sep 16 12:08:09.065470 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:09.065256    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pw7p9"]
Sep 16 12:08:09.766155 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:09.766122    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pw7p9" event={"ID":"e71d01c7-f1b2-44e0-b69d-94cf24847b6d","Type":"ContainerStarted","Data":"d3a96b22e68c1d61a1ecd4c1f5c242d96eb12bee2782603d1504be6dc2aed91e"}
Sep 16 12:08:11.776281 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:11.776245    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pw7p9" event={"ID":"e71d01c7-f1b2-44e0-b69d-94cf24847b6d","Type":"ContainerStarted","Data":"fd9322339638708959ebaa22cb35cd2dcf239be8148490acf0a2659bf882e0b2"}
Sep 16 12:08:11.776725 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:11.776294    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pw7p9" event={"ID":"e71d01c7-f1b2-44e0-b69d-94cf24847b6d","Type":"ContainerStarted","Data":"6c6cbbe2a5554cf7f8a48d8c04e4bbd5ff9bef401e3fbeb7ed4767cef39fcd41"}
Sep 16 12:08:11.793155 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:11.793108    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-pw7p9" podStartSLOduration=18.115528252 podStartE2EDuration="19.793094142s" totalImagesPullingTime="1.67756589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:07:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:11.792409677 +0000 UTC m=+285.493481002" watchObservedRunningTime="2026-09-16 12:08:11.793094142 +0000 UTC m=+285.494165469"
Sep 16 12:08:12.051866 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:12.051779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls\") pod \"dns-default-mlc2p\" (UID: \"096c53b4-6072-4b7e-9d0c-f2ffb8e13694\") " pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:08:12.054074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:12.054051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/096c53b4-6072-4b7e-9d0c-f2ffb8e13694-metrics-tls\") pod \"dns-default-mlc2p\" (UID: \"096c53b4-6072-4b7e-9d0c-f2ffb8e13694\") " pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:08:12.065817 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:12.065796    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7d4qr\""
Sep 16 12:08:12.074048 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:12.074033    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:08:12.343253 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:12.343221    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mlc2p"]
Sep 16 12:08:12.347764 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:08:12.347731    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096c53b4_6072_4b7e_9d0c_f2ffb8e13694.slice/crio-53b600761cc337f84b37c82f55fa94352afef1aab5422655181eb46ec643c96e WatchSource:0}: Error finding container 53b600761cc337f84b37c82f55fa94352afef1aab5422655181eb46ec643c96e: Status 404 returned error can't find the container with id 53b600761cc337f84b37c82f55fa94352afef1aab5422655181eb46ec643c96e
Sep 16 12:08:12.656878 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:12.656784    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85173723-783e-41a8-8b46-0459ca49e9e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:08:12.656878 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:12.656826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85173723-783e-41a8-8b46-0459ca49e9e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:08:12.657111 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:08:12.656963    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 12:08:12.657111 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:08:12.657038    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85173723-783e-41a8-8b46-0459ca49e9e1-insights-runtime-extractor-tls podName:85173723-783e-41a8-8b46-0459ca49e9e1 nodeName:}" failed. No retries permitted until 2026-09-16 12:08:28.657013282 +0000 UTC m=+302.358084588 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85173723-783e-41a8-8b46-0459ca49e9e1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sx7z6" (UID: "85173723-783e-41a8-8b46-0459ca49e9e1") : secret "insights-runtime-extractor-tls" not found
Sep 16 12:08:12.657318 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:12.657294    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85173723-783e-41a8-8b46-0459ca49e9e1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:08:12.781339 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:12.781299    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mlc2p" event={"ID":"096c53b4-6072-4b7e-9d0c-f2ffb8e13694","Type":"ContainerStarted","Data":"53b600761cc337f84b37c82f55fa94352afef1aab5422655181eb46ec643c96e"}
Sep 16 12:08:14.788434 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:14.788399    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mlc2p" event={"ID":"096c53b4-6072-4b7e-9d0c-f2ffb8e13694","Type":"ContainerStarted","Data":"12af789fc1e49debba4a52842bdfe22c839bd85bff63e307a00d938991dac934"}
Sep 16 12:08:14.788837 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:14.788456    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mlc2p" event={"ID":"096c53b4-6072-4b7e-9d0c-f2ffb8e13694","Type":"ContainerStarted","Data":"c5750d9ca75b66a30c2848b2a83e6f72d9e987c71a57a6fc2ab533f028b7b92f"}
Sep 16 12:08:14.788837 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:14.788542    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:08:14.811886 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:14.811836    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mlc2p" podStartSLOduration=251.385930728 podStartE2EDuration="4m12.811821533s" totalImagesPullingTime="1.425890805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:04:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:14.811521458 +0000 UTC m=+288.512592784" watchObservedRunningTime="2026-09-16 12:08:14.811821533 +0000 UTC m=+288.512892858"
Sep 16 12:08:23.158106 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:23.158076    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-zpqj5"]
Sep 16 12:08:23.161010 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:23.160994    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-zpqj5"
Sep 16 12:08:23.163498 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:23.163475    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 12:08:23.163624 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:23.163596    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 12:08:23.163624 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:23.163612    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-9dqt7\""
Sep 16 12:08:23.171409 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:23.171383    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-zpqj5"]
Sep 16 12:08:23.335926 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:23.335882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nscfn\" (UniqueName: \"kubernetes.io/projected/8bdb55fb-457e-4917-8168-53b1a0695c89-kube-api-access-nscfn\") pod \"downloads-76f99584f4-zpqj5\" (UID: \"8bdb55fb-457e-4917-8168-53b1a0695c89\") " pod="openshift-console/downloads-76f99584f4-zpqj5"
Sep 16 12:08:23.436470 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:23.436382    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nscfn\" (UniqueName: \"kubernetes.io/projected/8bdb55fb-457e-4917-8168-53b1a0695c89-kube-api-access-nscfn\") pod \"downloads-76f99584f4-zpqj5\" (UID: \"8bdb55fb-457e-4917-8168-53b1a0695c89\") " pod="openshift-console/downloads-76f99584f4-zpqj5"
Sep 16 12:08:23.445421 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:23.445379    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nscfn\" (UniqueName: \"kubernetes.io/projected/8bdb55fb-457e-4917-8168-53b1a0695c89-kube-api-access-nscfn\") pod \"downloads-76f99584f4-zpqj5\" (UID: \"8bdb55fb-457e-4917-8168-53b1a0695c89\") " pod="openshift-console/downloads-76f99584f4-zpqj5"
Sep 16 12:08:23.470026 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:23.469993    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-zpqj5"
Sep 16 12:08:23.740251 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:23.740172    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-zpqj5"]
Sep 16 12:08:23.743990 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:08:23.743959    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bdb55fb_457e_4917_8168_53b1a0695c89.slice/crio-22252fa9482743830772ca6590bcbe0fb5f042eca7a995746213dd269721696e WatchSource:0}: Error finding container 22252fa9482743830772ca6590bcbe0fb5f042eca7a995746213dd269721696e: Status 404 returned error can't find the container with id 22252fa9482743830772ca6590bcbe0fb5f042eca7a995746213dd269721696e
Sep 16 12:08:23.775703 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:08:23.775635    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-7cn45" podUID="5441211d-70c6-473a-ac90-417bd7aa57ec"
Sep 16 12:08:23.815400 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:23.815369    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-zpqj5" event={"ID":"8bdb55fb-457e-4917-8168-53b1a0695c89","Type":"ContainerStarted","Data":"22252fa9482743830772ca6590bcbe0fb5f042eca7a995746213dd269721696e"}
Sep 16 12:08:24.794069 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:24.794038    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mlc2p"
Sep 16 12:08:25.826062 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:25.825972    2579 generic.go:356] "Generic (PLEG): container finished" podID="28002685-3152-4c01-9543-56f847a1d212" containerID="19eef8ebe6cbbbc66d72307352164a3c10dc3c80390f02f5d59f045e67444f8d" exitCode=255
Sep 16 12:08:25.826062 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:25.826025    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86c756bfb-c7t5c" event={"ID":"28002685-3152-4c01-9543-56f847a1d212","Type":"ContainerDied","Data":"19eef8ebe6cbbbc66d72307352164a3c10dc3c80390f02f5d59f045e67444f8d"}
Sep 16 12:08:25.826560 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:25.826071    2579 scope.go:117] "RemoveContainer" containerID="8cfa8fa132823bd43bfe3b5010753105b2e9953108a6121f66fd4bc399f54b63"
Sep 16 12:08:25.833419 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:25.833397    2579 scope.go:117] "RemoveContainer" containerID="19eef8ebe6cbbbc66d72307352164a3c10dc3c80390f02f5d59f045e67444f8d"
Sep 16 12:08:25.833611 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:08:25.833585    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-86c756bfb-c7t5c_open-cluster-management-agent-addon(28002685-3152-4c01-9543-56f847a1d212)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86c756bfb-c7t5c" podUID="28002685-3152-4c01-9543-56f847a1d212"
Sep 16 12:08:26.674815 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:26.674778    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 12:08:27.412897 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:27.412860    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xctwt"]
Sep 16 12:08:27.416160 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:27.416141    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xctwt"
Sep 16 12:08:27.418471 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:27.418450    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 12:08:27.418634 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:27.418615    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-48nwm\""
Sep 16 12:08:27.423227 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:27.423010    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xctwt"]
Sep 16 12:08:27.567229 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:27.567188    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/40c34b1f-ed4a-4671-92c2-b27588a691c0-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-xctwt\" (UID: \"40c34b1f-ed4a-4671-92c2-b27588a691c0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xctwt"
Sep 16 12:08:27.667893 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:27.667813    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/40c34b1f-ed4a-4671-92c2-b27588a691c0-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-xctwt\" (UID: \"40c34b1f-ed4a-4671-92c2-b27588a691c0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xctwt"
Sep 16 12:08:27.668082 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:08:27.667976    2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 12:08:27.668082 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:08:27.668047    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40c34b1f-ed4a-4671-92c2-b27588a691c0-tls-certificates podName:40c34b1f-ed4a-4671-92c2-b27588a691c0 nodeName:}" failed. No retries permitted until 2026-09-16 12:08:28.168024293 +0000 UTC m=+301.869095611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/40c34b1f-ed4a-4671-92c2-b27588a691c0-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-xctwt" (UID: "40c34b1f-ed4a-4671-92c2-b27588a691c0") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 12:08:28.173075 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:28.173033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/40c34b1f-ed4a-4671-92c2-b27588a691c0-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-xctwt\" (UID: \"40c34b1f-ed4a-4671-92c2-b27588a691c0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xctwt"
Sep 16 12:08:28.173257 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:08:28.173192    2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 12:08:28.173304 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:08:28.173259    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40c34b1f-ed4a-4671-92c2-b27588a691c0-tls-certificates podName:40c34b1f-ed4a-4671-92c2-b27588a691c0 nodeName:}" failed. No retries permitted until 2026-09-16 12:08:29.173241601 +0000 UTC m=+302.874312906 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/40c34b1f-ed4a-4671-92c2-b27588a691c0-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-xctwt" (UID: "40c34b1f-ed4a-4671-92c2-b27588a691c0") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 12:08:28.678336 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:28.678302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85173723-783e-41a8-8b46-0459ca49e9e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:08:28.681200 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:28.681172    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85173723-783e-41a8-8b46-0459ca49e9e1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sx7z6\" (UID: \"85173723-783e-41a8-8b46-0459ca49e9e1\") " pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:08:28.686284 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:28.686234    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7hqfk\""
Sep 16 12:08:28.694145 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:28.694121    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sx7z6"
Sep 16 12:08:28.841561 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:28.841517    2579 generic.go:356] "Generic (PLEG): container finished" podID="d227c519-3639-481a-bda3-280eb42dc754" containerID="6d20a724787c386c94a908fa5278923783c0348c1b4e9fe8271b2970cc6460e0" exitCode=1
Sep 16 12:08:28.841727 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:28.841563    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt" event={"ID":"d227c519-3639-481a-bda3-280eb42dc754","Type":"ContainerDied","Data":"6d20a724787c386c94a908fa5278923783c0348c1b4e9fe8271b2970cc6460e0"}
Sep 16 12:08:28.841727 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:28.841621    2579 scope.go:117] "RemoveContainer" containerID="91e97bca08073a099cedc7be349263067ee85d542a99d0030f6e317cdf23e3cd"
Sep 16 12:08:28.842013 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:28.841994    2579 scope.go:117] "RemoveContainer" containerID="6d20a724787c386c94a908fa5278923783c0348c1b4e9fe8271b2970cc6460e0"
Sep 16 12:08:28.842239 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:08:28.842216    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-846f59685d-rk5qt_open-cluster-management-agent-addon(d227c519-3639-481a-bda3-280eb42dc754)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt" podUID="d227c519-3639-481a-bda3-280eb42dc754"
Sep 16 12:08:28.971867 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:28.971832    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sx7z6"]
Sep 16 12:08:28.975009 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:08:28.974977    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85173723_783e_41a8_8b46_0459ca49e9e1.slice/crio-47c712a2b92d2d8c149c4f756aab12f246913b47fcbcd8cb73107c19a3b1d3a5 WatchSource:0}: Error finding container 47c712a2b92d2d8c149c4f756aab12f246913b47fcbcd8cb73107c19a3b1d3a5: Status 404 returned error can't find the container with id 47c712a2b92d2d8c149c4f756aab12f246913b47fcbcd8cb73107c19a3b1d3a5
Sep 16 12:08:29.033241 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:29.033222    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:08:29.183060 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:29.182956    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/40c34b1f-ed4a-4671-92c2-b27588a691c0-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-xctwt\" (UID: \"40c34b1f-ed4a-4671-92c2-b27588a691c0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xctwt"
Sep 16 12:08:29.185851 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:29.185804    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/40c34b1f-ed4a-4671-92c2-b27588a691c0-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-xctwt\" (UID: \"40c34b1f-ed4a-4671-92c2-b27588a691c0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xctwt"
Sep 16 12:08:29.226728 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:29.226691    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xctwt"
Sep 16 12:08:29.463150 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:29.463073    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt"
Sep 16 12:08:29.511558 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:29.511517    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xctwt"]
Sep 16 12:08:29.514984 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:08:29.514949    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c34b1f_ed4a_4671_92c2_b27588a691c0.slice/crio-0099961afa66ca5968516bdb8d5db476ae26edfe2580654b3db8cae007555af4 WatchSource:0}: Error finding container 0099961afa66ca5968516bdb8d5db476ae26edfe2580654b3db8cae007555af4: Status 404 returned error can't find the container with id 0099961afa66ca5968516bdb8d5db476ae26edfe2580654b3db8cae007555af4
Sep 16 12:08:29.847022 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:29.846987    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xctwt" event={"ID":"40c34b1f-ed4a-4671-92c2-b27588a691c0","Type":"ContainerStarted","Data":"0099961afa66ca5968516bdb8d5db476ae26edfe2580654b3db8cae007555af4"}
Sep 16 12:08:29.849329 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:29.849294    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sx7z6" event={"ID":"85173723-783e-41a8-8b46-0459ca49e9e1","Type":"ContainerStarted","Data":"3e4280f2ce033c49ed79f054c7d33bdca6d9808ff4724a4399791185783392b5"}
Sep 16 12:08:29.849458 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:29.849353    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sx7z6" event={"ID":"85173723-783e-41a8-8b46-0459ca49e9e1","Type":"ContainerStarted","Data":"47c712a2b92d2d8c149c4f756aab12f246913b47fcbcd8cb73107c19a3b1d3a5"}
Sep 16 12:08:29.851311 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:29.851290    2579 scope.go:117] "RemoveContainer" containerID="6d20a724787c386c94a908fa5278923783c0348c1b4e9fe8271b2970cc6460e0"
Sep 16 12:08:29.851523 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:08:29.851492    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-846f59685d-rk5qt_open-cluster-management-agent-addon(d227c519-3639-481a-bda3-280eb42dc754)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt" podUID="d227c519-3639-481a-bda3-280eb42dc754"
Sep 16 12:08:30.857326 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:30.857284    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xctwt" event={"ID":"40c34b1f-ed4a-4671-92c2-b27588a691c0","Type":"ContainerStarted","Data":"031ca366813c416f170af96b1dd839a0d62c103a9a1cdb3f83fde7f97bff990a"}
Sep 16 12:08:30.857878 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:30.857865    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xctwt"
Sep 16 12:08:30.859241 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:30.859201    2579 patch_prober.go:28] interesting pod/prometheus-operator-admission-webhook-5864cd99d7-xctwt container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.17:8443/healthz\": dial tcp 10.134.0.17:8443: connect: connection refused" start-of-body=
Sep 16 12:08:30.859383 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:30.859249    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xctwt" podUID="40c34b1f-ed4a-4671-92c2-b27588a691c0" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.134.0.17:8443/healthz\": dial tcp 10.134.0.17:8443: connect: connection refused"
Sep 16 12:08:30.860209 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:30.860181    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sx7z6" event={"ID":"85173723-783e-41a8-8b46-0459ca49e9e1","Type":"ContainerStarted","Data":"f00be1c1e13a34f5519d7c9e702341e49cb555380f1b6ee21bbbdb2e8b480586"}
Sep 16 12:08:30.873584 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:30.873311    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xctwt" podStartSLOduration=2.622471707 podStartE2EDuration="3.873290106s" totalImagesPullingTime="1.250818399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:30.873249777 +0000 UTC m=+304.574321107" watchObservedRunningTime="2026-09-16 12:08:30.873290106 +0000 UTC m=+304.574361410"
Sep 16 12:08:31.872604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:31.872570    2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt"
Sep 16 12:08:31.873090 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:31.872964    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-xctwt"
Sep 16 12:08:31.873090 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:31.873047    2579 scope.go:117] "RemoveContainer" containerID="6d20a724787c386c94a908fa5278923783c0348c1b4e9fe8271b2970cc6460e0"
Sep 16 12:08:31.873239 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:08:31.873212    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-846f59685d-rk5qt_open-cluster-management-agent-addon(d227c519-3639-481a-bda3-280eb42dc754)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt" podUID="d227c519-3639-481a-bda3-280eb42dc754"
Sep 16 12:08:31.889751 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:31.889721    2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86c756bfb-c7t5c"
Sep 16 12:08:31.897892 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:31.897866    2579 scope.go:117] "RemoveContainer" containerID="19eef8ebe6cbbbc66d72307352164a3c10dc3c80390f02f5d59f045e67444f8d"
Sep 16 12:08:31.898086 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:08:31.898060    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-86c756bfb-c7t5c_open-cluster-management-agent-addon(28002685-3152-4c01-9543-56f847a1d212)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86c756bfb-c7t5c" podUID="28002685-3152-4c01-9543-56f847a1d212"
Sep 16 12:08:32.871190 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:32.871149    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sx7z6" event={"ID":"85173723-783e-41a8-8b46-0459ca49e9e1","Type":"ContainerStarted","Data":"e181c5467d199d37403f920a504adba8c2bad0800f9f97446855df3d6f0ed17e"}
Sep 16 12:08:32.890102 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:32.890019    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sx7z6" podStartSLOduration=33.947238035 podStartE2EDuration="36.889995495s" totalImagesPullingTime="2.94275746s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:07:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:32.888747342 +0000 UTC m=+306.589818671" watchObservedRunningTime="2026-09-16 12:08:32.889995495 +0000 UTC m=+306.591066818"
Sep 16 12:08:36.776160 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:36.776124    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:08:37.862340 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:37.862301    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n2r6c"]
Sep 16 12:08:37.865620 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:37.865597    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:37.868501 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:37.868474    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 12:08:37.868501 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:37.868497    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rvlt5\""
Sep 16 12:08:37.868700 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:37.868502    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 12:08:37.869503 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:37.869478    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 12:08:37.869632 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:37.869605    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 12:08:37.869757 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:37.869605    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 12:08:37.869757 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:37.869684    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 12:08:37.960818 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:37.960781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ae8d233-edab-4270-b326-6fc1277bfb20-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:37.960977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:37.960830    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7kvm\" (UniqueName: \"kubernetes.io/projected/8ae8d233-edab-4270-b326-6fc1277bfb20-kube-api-access-p7kvm\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:37.960977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:37.960894    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ae8d233-edab-4270-b326-6fc1277bfb20-metrics-client-ca\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:37.960977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:37.960937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ae8d233-edab-4270-b326-6fc1277bfb20-node-exporter-tls\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:37.960977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:37.960962    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ae8d233-edab-4270-b326-6fc1277bfb20-node-exporter-accelerators-collector-config\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:37.961210 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:37.961058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ae8d233-edab-4270-b326-6fc1277bfb20-sys\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:37.961210 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:37.961102    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ae8d233-edab-4270-b326-6fc1277bfb20-root\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:37.961210 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:37.961136    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ae8d233-edab-4270-b326-6fc1277bfb20-node-exporter-textfile\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:37.961210 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:37.961192    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ae8d233-edab-4270-b326-6fc1277bfb20-node-exporter-wtmp\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:38.062526 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.062476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ae8d233-edab-4270-b326-6fc1277bfb20-metrics-client-ca\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:38.062730 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.062551    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ae8d233-edab-4270-b326-6fc1277bfb20-node-exporter-tls\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:38.062730 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.062577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ae8d233-edab-4270-b326-6fc1277bfb20-node-exporter-accelerators-collector-config\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:38.062730 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.062630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ae8d233-edab-4270-b326-6fc1277bfb20-sys\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:38.062730 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.062679    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ae8d233-edab-4270-b326-6fc1277bfb20-root\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:38.062730 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.062712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ae8d233-edab-4270-b326-6fc1277bfb20-node-exporter-textfile\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:38.062983 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.062841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ae8d233-edab-4270-b326-6fc1277bfb20-sys\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:38.062983 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.062852    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ae8d233-edab-4270-b326-6fc1277bfb20-root\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:38.062983 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.062877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ae8d233-edab-4270-b326-6fc1277bfb20-node-exporter-wtmp\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:38.062983 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.062941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ae8d233-edab-4270-b326-6fc1277bfb20-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:38.062983 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.062978    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7kvm\" (UniqueName: \"kubernetes.io/projected/8ae8d233-edab-4270-b326-6fc1277bfb20-kube-api-access-p7kvm\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:38.063224 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.063058    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ae8d233-edab-4270-b326-6fc1277bfb20-node-exporter-wtmp\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:38.063274 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.063234    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ae8d233-edab-4270-b326-6fc1277bfb20-node-exporter-textfile\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:38.063401 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.063354    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ae8d233-edab-4270-b326-6fc1277bfb20-node-exporter-accelerators-collector-config\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:38.063519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.063429    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ae8d233-edab-4270-b326-6fc1277bfb20-metrics-client-ca\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:38.065526 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.065502    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ae8d233-edab-4270-b326-6fc1277bfb20-node-exporter-tls\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:38.065885 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.065862    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ae8d233-edab-4270-b326-6fc1277bfb20-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:38.076257 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.076235    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7kvm\" (UniqueName: \"kubernetes.io/projected/8ae8d233-edab-4270-b326-6fc1277bfb20-kube-api-access-p7kvm\") pod \"node-exporter-n2r6c\" (UID: \"8ae8d233-edab-4270-b326-6fc1277bfb20\") " pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:38.186928 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.186852    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n2r6c"
Sep 16 12:08:38.945923 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.945886    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:08:38.948838 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.948814    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:38.951768 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.951687    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 12:08:38.951768 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.951737    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 12:08:38.951998 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.951797    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 12:08:38.951998 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.951871    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 12:08:38.951998 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.951907    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 12:08:38.951998 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.951746    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 12:08:38.952223 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.952021    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 12:08:38.952223 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.952053    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 12:08:38.952223 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.952059    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 12:08:38.952223 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.951691    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7cwd5\""
Sep 16 12:08:38.964815 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.964782    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:08:38.970527 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.970495    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-config-volume\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:38.970641 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.970533    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-web-config\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:38.970641 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.970565    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:38.970641 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.970600    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:38.970814 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.970697    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:38.970814 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.970732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9jvh\" (UniqueName: \"kubernetes.io/projected/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-kube-api-access-z9jvh\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:38.970814 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.970761    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:38.970814 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.970799    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:38.971016 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.970848    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:38.971016 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.970880    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:38.971016 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.970909    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:38.971016 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.970942    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-config-out\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:38.971016 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:38.970988    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.071561 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.071533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.071561 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.071563    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.071830 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.071592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.071830 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.071623    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-config-out\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.071830 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.071688    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.071830 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.071724    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-config-volume\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.071830 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.071749    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-web-config\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.071830 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.071785    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.071830 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.071819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.072181 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:08:39.071937    2579 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 16 12:08:39.072181 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:08:39.072008    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-main-tls podName:9dc098ac-4cd7-47c2-a026-e6e933e20f4f nodeName:}" failed. No retries permitted until 2026-09-16 12:08:39.571987545 +0000 UTC m=+313.273058866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "9dc098ac-4cd7-47c2-a026-e6e933e20f4f") : secret "alertmanager-main-tls" not found
Sep 16 12:08:39.072181 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.072030    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.072359 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.072250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.072359 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.072288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9jvh\" (UniqueName: \"kubernetes.io/projected/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-kube-api-access-z9jvh\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.072359 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.072325    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.072511 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.072368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.073635 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.072597    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.073635 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.073563    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.075920 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.074956    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-config-out\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.075920 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.074961    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.075920 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.075402    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-config-volume\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.075920 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.075544    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.075920 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.075836    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-web-config\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.075920 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.075912    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.076568 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.076421    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.077126 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.077104    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.081325 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.081302    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9jvh\" (UniqueName: \"kubernetes.io/projected/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-kube-api-access-z9jvh\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.577034 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.576999    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.580050 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.580019    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:39.860473 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:39.860438    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:08:40.174703 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:08:40.174670    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ae8d233_edab_4270_b326_6fc1277bfb20.slice/crio-2af16f04c95475ec5e41d8d7fd54c25d96aae821c8f171685611eec36d98badb WatchSource:0}: Error finding container 2af16f04c95475ec5e41d8d7fd54c25d96aae821c8f171685611eec36d98badb: Status 404 returned error can't find the container with id 2af16f04c95475ec5e41d8d7fd54c25d96aae821c8f171685611eec36d98badb
Sep 16 12:08:40.448185 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.448154    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:08:40.450840 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:08:40.450807    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc098ac_4cd7_47c2_a026_e6e933e20f4f.slice/crio-a9ac5f6aaec705e744bbd140865d7f8bd8a60e003f12ad9d7a973d416d0cb951 WatchSource:0}: Error finding container a9ac5f6aaec705e744bbd140865d7f8bd8a60e003f12ad9d7a973d416d0cb951: Status 404 returned error can't find the container with id a9ac5f6aaec705e744bbd140865d7f8bd8a60e003f12ad9d7a973d416d0cb951
Sep 16 12:08:40.661455 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.661413    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76bd8984d-9md9w"]
Sep 16 12:08:40.663969 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.663943    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:08:40.668045 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.667206    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 12:08:40.668045 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.667280    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 12:08:40.668045 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.667452    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 12:08:40.668045 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.667671    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 12:08:40.668045 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.667672    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 12:08:40.668395 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.668079    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-982qg\""
Sep 16 12:08:40.679108 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.679079    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76bd8984d-9md9w"]
Sep 16 12:08:40.681594 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.681573    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 12:08:40.686078 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.686054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-console-serving-cert\") pod \"console-76bd8984d-9md9w\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") " pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:08:40.686290 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.686257    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-service-ca\") pod \"console-76bd8984d-9md9w\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") " pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:08:40.686412 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.686302    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spmnx\" (UniqueName: \"kubernetes.io/projected/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-kube-api-access-spmnx\") pod \"console-76bd8984d-9md9w\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") " pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:08:40.686412 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.686340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-trusted-ca-bundle\") pod \"console-76bd8984d-9md9w\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") " pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:08:40.686412 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.686367    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-oauth-serving-cert\") pod \"console-76bd8984d-9md9w\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") " pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:08:40.686568 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.686426    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-console-config\") pod \"console-76bd8984d-9md9w\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") " pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:08:40.686568 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.686458    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-console-oauth-config\") pod \"console-76bd8984d-9md9w\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") " pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:08:40.787571 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.787489    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-console-config\") pod \"console-76bd8984d-9md9w\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") " pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:08:40.787571 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.787542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-console-oauth-config\") pod \"console-76bd8984d-9md9w\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") " pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:08:40.787827 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.787584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-console-serving-cert\") pod \"console-76bd8984d-9md9w\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") " pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:08:40.787827 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.787616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-service-ca\") pod \"console-76bd8984d-9md9w\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") " pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:08:40.787827 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.787647    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spmnx\" (UniqueName: \"kubernetes.io/projected/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-kube-api-access-spmnx\") pod \"console-76bd8984d-9md9w\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") " pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:08:40.787827 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.787710    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-trusted-ca-bundle\") pod \"console-76bd8984d-9md9w\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") " pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:08:40.787827 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.787740    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-oauth-serving-cert\") pod \"console-76bd8984d-9md9w\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") " pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:08:40.788506 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.788480    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-oauth-serving-cert\") pod \"console-76bd8984d-9md9w\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") " pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:08:40.789097 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.789062    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-console-config\") pod \"console-76bd8984d-9md9w\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") " pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:08:40.789790 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.789744    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-service-ca\") pod \"console-76bd8984d-9md9w\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") " pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:08:40.790148 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.790101    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-trusted-ca-bundle\") pod \"console-76bd8984d-9md9w\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") " pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:08:40.794736 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.794683    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-console-oauth-config\") pod \"console-76bd8984d-9md9w\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") " pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:08:40.796023 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.795980    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-console-serving-cert\") pod \"console-76bd8984d-9md9w\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") " pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:08:40.798031 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.797958    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spmnx\" (UniqueName: \"kubernetes.io/projected/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-kube-api-access-spmnx\") pod \"console-76bd8984d-9md9w\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") " pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:08:40.903766 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.903215    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-zpqj5" event={"ID":"8bdb55fb-457e-4917-8168-53b1a0695c89","Type":"ContainerStarted","Data":"c45d64b66d32efe049334a91e0daf9982b1b21cdfeb2193a666f8ae9e66f303e"}
Sep 16 12:08:40.903766 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.903706    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-zpqj5"
Sep 16 12:08:40.906318 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.906255    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n2r6c" event={"ID":"8ae8d233-edab-4270-b326-6fc1277bfb20","Type":"ContainerStarted","Data":"2af16f04c95475ec5e41d8d7fd54c25d96aae821c8f171685611eec36d98badb"}
Sep 16 12:08:40.907951 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.907871    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9dc098ac-4cd7-47c2-a026-e6e933e20f4f","Type":"ContainerStarted","Data":"a9ac5f6aaec705e744bbd140865d7f8bd8a60e003f12ad9d7a973d416d0cb951"}
Sep 16 12:08:40.917868 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.917833    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-zpqj5"
Sep 16 12:08:40.922533 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.922453    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-zpqj5" podStartSLOduration=1.421436875 podStartE2EDuration="17.922430482s" totalImagesPullingTime="16.500993607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:40.920758382 +0000 UTC m=+314.621829708" watchObservedRunningTime="2026-09-16 12:08:40.922430482 +0000 UTC m=+314.623501809"
Sep 16 12:08:40.985838 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:40.985752    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:08:41.394149 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:41.394009    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76bd8984d-9md9w"]
Sep 16 12:08:41.913224 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:41.913191    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76bd8984d-9md9w" event={"ID":"ed5ab3f0-c343-47c0-9695-ebcbed31ec77","Type":"ContainerStarted","Data":"54f29443782e45f5e2c75b47be5ab962a5d732d061e8f1107181c273cd972edd"}
Sep 16 12:08:41.915183 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:41.915141    2579 generic.go:356] "Generic (PLEG): container finished" podID="8ae8d233-edab-4270-b326-6fc1277bfb20" containerID="dfdd490fd87584a7ebb32f17957bd661727559db2b3e74c6613d212b7b3f9b46" exitCode=0
Sep 16 12:08:41.915296 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:41.915225    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n2r6c" event={"ID":"8ae8d233-edab-4270-b326-6fc1277bfb20","Type":"ContainerDied","Data":"dfdd490fd87584a7ebb32f17957bd661727559db2b3e74c6613d212b7b3f9b46"}
Sep 16 12:08:42.225805 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.225733    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-dc766c59d-hfznn"]
Sep 16 12:08:42.231735 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.231711    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:08:42.234980 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.234917    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 12:08:42.235111 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.235026    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 12:08:42.235264 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.235237    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-j5zp9\""
Sep 16 12:08:42.235337 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.235267    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 12:08:42.235578 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.235417    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-b23cev6i5aafv\""
Sep 16 12:08:42.235970 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.235943    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 12:08:42.242669 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.242600    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-dc766c59d-hfznn"]
Sep 16 12:08:42.302360 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.302317    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/02da3487-ee2b-4d3d-9b0a-8414a6c0c653-client-ca-bundle\") pod \"metrics-server-dc766c59d-hfznn\" (UID: \"02da3487-ee2b-4d3d-9b0a-8414a6c0c653\") " pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:08:42.302565 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.302373    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02da3487-ee2b-4d3d-9b0a-8414a6c0c653-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dc766c59d-hfznn\" (UID: \"02da3487-ee2b-4d3d-9b0a-8414a6c0c653\") " pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:08:42.302565 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.302446    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/02da3487-ee2b-4d3d-9b0a-8414a6c0c653-secret-metrics-server-tls\") pod \"metrics-server-dc766c59d-hfznn\" (UID: \"02da3487-ee2b-4d3d-9b0a-8414a6c0c653\") " pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:08:42.302565 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.302477    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/02da3487-ee2b-4d3d-9b0a-8414a6c0c653-secret-metrics-server-client-certs\") pod \"metrics-server-dc766c59d-hfznn\" (UID: \"02da3487-ee2b-4d3d-9b0a-8414a6c0c653\") " pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:08:42.302565 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.302550    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/02da3487-ee2b-4d3d-9b0a-8414a6c0c653-metrics-server-audit-profiles\") pod \"metrics-server-dc766c59d-hfznn\" (UID: \"02da3487-ee2b-4d3d-9b0a-8414a6c0c653\") " pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:08:42.302834 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.302634    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f472d\" (UniqueName: \"kubernetes.io/projected/02da3487-ee2b-4d3d-9b0a-8414a6c0c653-kube-api-access-f472d\") pod \"metrics-server-dc766c59d-hfznn\" (UID: \"02da3487-ee2b-4d3d-9b0a-8414a6c0c653\") " pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:08:42.302834 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.302729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/02da3487-ee2b-4d3d-9b0a-8414a6c0c653-audit-log\") pod \"metrics-server-dc766c59d-hfznn\" (UID: \"02da3487-ee2b-4d3d-9b0a-8414a6c0c653\") " pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:08:42.403515 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.403472    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/02da3487-ee2b-4d3d-9b0a-8414a6c0c653-metrics-server-audit-profiles\") pod \"metrics-server-dc766c59d-hfznn\" (UID: \"02da3487-ee2b-4d3d-9b0a-8414a6c0c653\") " pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:08:42.404077 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.403535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f472d\" (UniqueName: \"kubernetes.io/projected/02da3487-ee2b-4d3d-9b0a-8414a6c0c653-kube-api-access-f472d\") pod \"metrics-server-dc766c59d-hfznn\" (UID: \"02da3487-ee2b-4d3d-9b0a-8414a6c0c653\") " pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:08:42.404077 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.403571    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/02da3487-ee2b-4d3d-9b0a-8414a6c0c653-audit-log\") pod \"metrics-server-dc766c59d-hfznn\" (UID: \"02da3487-ee2b-4d3d-9b0a-8414a6c0c653\") " pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:08:42.404077 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.403634    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/02da3487-ee2b-4d3d-9b0a-8414a6c0c653-client-ca-bundle\") pod \"metrics-server-dc766c59d-hfznn\" (UID: \"02da3487-ee2b-4d3d-9b0a-8414a6c0c653\") " pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:08:42.404077 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.403677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02da3487-ee2b-4d3d-9b0a-8414a6c0c653-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dc766c59d-hfznn\" (UID: \"02da3487-ee2b-4d3d-9b0a-8414a6c0c653\") " pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:08:42.404077 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.403728    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/02da3487-ee2b-4d3d-9b0a-8414a6c0c653-secret-metrics-server-tls\") pod \"metrics-server-dc766c59d-hfznn\" (UID: \"02da3487-ee2b-4d3d-9b0a-8414a6c0c653\") " pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:08:42.404077 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.403761    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/02da3487-ee2b-4d3d-9b0a-8414a6c0c653-secret-metrics-server-client-certs\") pod \"metrics-server-dc766c59d-hfznn\" (UID: \"02da3487-ee2b-4d3d-9b0a-8414a6c0c653\") " pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:08:42.405299 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.405270    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02da3487-ee2b-4d3d-9b0a-8414a6c0c653-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-dc766c59d-hfznn\" (UID: \"02da3487-ee2b-4d3d-9b0a-8414a6c0c653\") " pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:08:42.406652 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.406625    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/02da3487-ee2b-4d3d-9b0a-8414a6c0c653-metrics-server-audit-profiles\") pod \"metrics-server-dc766c59d-hfznn\" (UID: \"02da3487-ee2b-4d3d-9b0a-8414a6c0c653\") " pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:08:42.406861 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.406839    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/02da3487-ee2b-4d3d-9b0a-8414a6c0c653-audit-log\") pod \"metrics-server-dc766c59d-hfznn\" (UID: \"02da3487-ee2b-4d3d-9b0a-8414a6c0c653\") " pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:08:42.409021 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.408973    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/02da3487-ee2b-4d3d-9b0a-8414a6c0c653-secret-metrics-server-tls\") pod \"metrics-server-dc766c59d-hfznn\" (UID: \"02da3487-ee2b-4d3d-9b0a-8414a6c0c653\") " pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:08:42.409590 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.409546    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/02da3487-ee2b-4d3d-9b0a-8414a6c0c653-secret-metrics-server-client-certs\") pod \"metrics-server-dc766c59d-hfznn\" (UID: \"02da3487-ee2b-4d3d-9b0a-8414a6c0c653\") " pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:08:42.409750 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.409726    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/02da3487-ee2b-4d3d-9b0a-8414a6c0c653-client-ca-bundle\") pod \"metrics-server-dc766c59d-hfznn\" (UID: \"02da3487-ee2b-4d3d-9b0a-8414a6c0c653\") " pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:08:42.415433 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.415395    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f472d\" (UniqueName: \"kubernetes.io/projected/02da3487-ee2b-4d3d-9b0a-8414a6c0c653-kube-api-access-f472d\") pod \"metrics-server-dc766c59d-hfznn\" (UID: \"02da3487-ee2b-4d3d-9b0a-8414a6c0c653\") " pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:08:42.546035 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.545939    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:08:42.860805 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.860771    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-dc766c59d-hfznn"]
Sep 16 12:08:42.862978 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:08:42.862949    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02da3487_ee2b_4d3d_9b0a_8414a6c0c653.slice/crio-eb6c803f0557b46401dfafbab80fbd5c5e4df5d9efbb45e9587e8b06529f83c7 WatchSource:0}: Error finding container eb6c803f0557b46401dfafbab80fbd5c5e4df5d9efbb45e9587e8b06529f83c7: Status 404 returned error can't find the container with id eb6c803f0557b46401dfafbab80fbd5c5e4df5d9efbb45e9587e8b06529f83c7
Sep 16 12:08:42.924086 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.923935    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n2r6c" event={"ID":"8ae8d233-edab-4270-b326-6fc1277bfb20","Type":"ContainerStarted","Data":"a7bc5907e3a2f01dada1886d1de8adb5152a605ce3aaaa3fe4e6bc110dfbbf8d"}
Sep 16 12:08:42.924086 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.924002    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n2r6c" event={"ID":"8ae8d233-edab-4270-b326-6fc1277bfb20","Type":"ContainerStarted","Data":"9cd2b43cd697b3413a8194d9b503ee917b542e87f95566fd3bb971e9cec726c6"}
Sep 16 12:08:42.926932 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.926867    2579 generic.go:356] "Generic (PLEG): container finished" podID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerID="2c71954d27540bdd08c0af9995784272353b79004822d87bd7154f3a9a3f4aba" exitCode=0
Sep 16 12:08:42.927158 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.926990    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9dc098ac-4cd7-47c2-a026-e6e933e20f4f","Type":"ContainerDied","Data":"2c71954d27540bdd08c0af9995784272353b79004822d87bd7154f3a9a3f4aba"}
Sep 16 12:08:42.929771 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.929600    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-dc766c59d-hfznn" event={"ID":"02da3487-ee2b-4d3d-9b0a-8414a6c0c653","Type":"ContainerStarted","Data":"eb6c803f0557b46401dfafbab80fbd5c5e4df5d9efbb45e9587e8b06529f83c7"}
Sep 16 12:08:42.947988 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:42.947921    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n2r6c" podStartSLOduration=4.913883241 podStartE2EDuration="5.94790039s" totalImagesPullingTime="950.574224ms" totalInitContainerRuntime="83.442925ms" isStatefulPod=true podCreationTimestamp="2026-09-16 12:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:42.946348323 +0000 UTC m=+316.647419651" watchObservedRunningTime="2026-09-16 12:08:42.94790039 +0000 UTC m=+316.648971717"
Sep 16 12:08:46.777125 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:46.777092    2579 scope.go:117] "RemoveContainer" containerID="6d20a724787c386c94a908fa5278923783c0348c1b4e9fe8271b2970cc6460e0"
Sep 16 12:08:46.784778 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:46.784752    2579 scope.go:117] "RemoveContainer" containerID="19eef8ebe6cbbbc66d72307352164a3c10dc3c80390f02f5d59f045e67444f8d"
Sep 16 12:08:47.475826 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:47.475720    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76bd8984d-9md9w"]
Sep 16 12:08:47.962440 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:47.962176    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-dc766c59d-hfznn" event={"ID":"02da3487-ee2b-4d3d-9b0a-8414a6c0c653","Type":"ContainerStarted","Data":"89781e7f03338a6939d91dde30c3c69c122fc44c83b92a7766e7a81cb4cf3666"}
Sep 16 12:08:47.964857 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:47.964820    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-86c756bfb-c7t5c" event={"ID":"28002685-3152-4c01-9543-56f847a1d212","Type":"ContainerStarted","Data":"319b34144fe637354b8a1bd13ed3b82ab6989a9de9635f4c65b288706fc94200"}
Sep 16 12:08:47.966849 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:47.966820    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76bd8984d-9md9w" event={"ID":"ed5ab3f0-c343-47c0-9695-ebcbed31ec77","Type":"ContainerStarted","Data":"608597662f6731a01c762e4aeb3b946b31a9773ccbb82fd2b4b0d9dcb713ffbb"}
Sep 16 12:08:47.968996 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:47.968968    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt" event={"ID":"d227c519-3639-481a-bda3-280eb42dc754","Type":"ContainerStarted","Data":"7dbd5394fdbaa9732f18e2677609c128b6436a4ddd6903eae28ad04c76bbe47b"}
Sep 16 12:08:47.969411 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:47.969392    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt"
Sep 16 12:08:47.971259 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:47.971240    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-846f59685d-rk5qt"
Sep 16 12:08:47.973085 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:47.973063    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9dc098ac-4cd7-47c2-a026-e6e933e20f4f","Type":"ContainerStarted","Data":"8c20c00f59afb8cc2cfabedfa6cad362c8c46d123ab0cc651bf024ab94d10e6a"}
Sep 16 12:08:47.973198 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:47.973128    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9dc098ac-4cd7-47c2-a026-e6e933e20f4f","Type":"ContainerStarted","Data":"8b496967ebd0664b497ac532fbe84103b74388a5c8e2ff520b283c7e6908acb7"}
Sep 16 12:08:47.973198 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:47.973175    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9dc098ac-4cd7-47c2-a026-e6e933e20f4f","Type":"ContainerStarted","Data":"3248f430b8759cde9f4c31b0210825e560a785d13852c802856f8442b6e5d747"}
Sep 16 12:08:47.973301 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:47.973218    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9dc098ac-4cd7-47c2-a026-e6e933e20f4f","Type":"ContainerStarted","Data":"b6cfef3f3b52d5141fe22f151542dba259dd3c9c3b44ff154a091d868656142b"}
Sep 16 12:08:47.973301 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:47.973257    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9dc098ac-4cd7-47c2-a026-e6e933e20f4f","Type":"ContainerStarted","Data":"baae7854507279469f63913b721421cea49f7fc8b4bd107af5ce3133630d4364"}
Sep 16 12:08:47.983283 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:47.983237    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-dc766c59d-hfznn" podStartSLOduration=1.9623113779999999 podStartE2EDuration="5.983220774s" totalImagesPullingTime="4.020909396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:47.981321831 +0000 UTC m=+321.682393148" watchObservedRunningTime="2026-09-16 12:08:47.983220774 +0000 UTC m=+321.684292100"
Sep 16 12:08:48.001738 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:48.001678    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76bd8984d-9md9w" podStartSLOduration=2.517416056 podStartE2EDuration="8.001649076s" totalImagesPullingTime="5.48423302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:48.000065828 +0000 UTC m=+321.701137180" watchObservedRunningTime="2026-09-16 12:08:48.001649076 +0000 UTC m=+321.702720402"
Sep 16 12:08:49.985209 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:49.985172    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9dc098ac-4cd7-47c2-a026-e6e933e20f4f","Type":"ContainerStarted","Data":"3a7c4ea08e4a2f89c1c2bf13c5109961d58546f72f686d85e3633ad4d3677365"}
Sep 16 12:08:50.014493 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:50.014440    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.022254422 podStartE2EDuration="12.014421814s" totalImagesPullingTime="6.969843407s" totalInitContainerRuntime="22.323985ms" isStatefulPod=false podCreationTimestamp="2026-09-16 12:08:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:50.013149606 +0000 UTC m=+323.714220934" watchObservedRunningTime="2026-09-16 12:08:50.014421814 +0000 UTC m=+323.715493139"
Sep 16 12:08:50.985928 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:50.985893    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:08:51.354624 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:51.354585    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76c88fdd8d-9dc4s"]
Sep 16 12:08:51.377288 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:51.377254    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:08:51.386574 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:51.386543    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76c88fdd8d-9dc4s"]
Sep 16 12:08:51.493501 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:51.493454    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-oauth-serving-cert\") pod \"console-76c88fdd8d-9dc4s\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") " pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:08:51.493501 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:51.493508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-console-oauth-config\") pod \"console-76c88fdd8d-9dc4s\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") " pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:08:51.493744 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:51.493528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-service-ca\") pod \"console-76c88fdd8d-9dc4s\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") " pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:08:51.493744 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:51.493598    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-console-serving-cert\") pod \"console-76c88fdd8d-9dc4s\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") " pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:08:51.493744 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:51.493643    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-console-config\") pod \"console-76c88fdd8d-9dc4s\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") " pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:08:51.493744 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:51.493689    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glgrn\" (UniqueName: \"kubernetes.io/projected/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-kube-api-access-glgrn\") pod \"console-76c88fdd8d-9dc4s\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") " pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:08:51.493744 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:51.493735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-trusted-ca-bundle\") pod \"console-76c88fdd8d-9dc4s\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") " pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:08:51.594954 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:51.594918    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-oauth-serving-cert\") pod \"console-76c88fdd8d-9dc4s\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") " pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:08:51.595072 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:51.594967    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-console-oauth-config\") pod \"console-76c88fdd8d-9dc4s\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") " pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:08:51.595072 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:51.594985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-service-ca\") pod \"console-76c88fdd8d-9dc4s\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") " pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:08:51.595072 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:51.595014    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-console-serving-cert\") pod \"console-76c88fdd8d-9dc4s\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") " pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:08:51.595072 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:51.595034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-console-config\") pod \"console-76c88fdd8d-9dc4s\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") " pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:08:51.595072 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:51.595049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glgrn\" (UniqueName: \"kubernetes.io/projected/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-kube-api-access-glgrn\") pod \"console-76c88fdd8d-9dc4s\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") " pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:08:51.595072 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:51.595069    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-trusted-ca-bundle\") pod \"console-76c88fdd8d-9dc4s\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") " pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:08:51.595888 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:51.595862    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-service-ca\") pod \"console-76c88fdd8d-9dc4s\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") " pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:08:51.596006 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:51.595893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-console-config\") pod \"console-76c88fdd8d-9dc4s\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") " pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:08:51.596006 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:51.595862    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-oauth-serving-cert\") pod \"console-76c88fdd8d-9dc4s\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") " pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:08:51.596160 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:51.596140    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-trusted-ca-bundle\") pod \"console-76c88fdd8d-9dc4s\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") " pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:08:51.597604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:51.597583    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-console-oauth-config\") pod \"console-76c88fdd8d-9dc4s\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") " pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:08:51.597664 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:51.597592    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-console-serving-cert\") pod \"console-76c88fdd8d-9dc4s\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") " pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:08:51.603911 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:51.603890    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glgrn\" (UniqueName: \"kubernetes.io/projected/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-kube-api-access-glgrn\") pod \"console-76c88fdd8d-9dc4s\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") " pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:08:51.687053 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:51.686955    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:08:52.000981 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:52.000955    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76c88fdd8d-9dc4s"]
Sep 16 12:08:52.005584 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:08:52.005552    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a3cb28_215a_41e1_86f9_bcc4c0923e7c.slice/crio-86f3f847aa31f15af7d74ad1a7719584b4fbc0ed4608b9adab02523677747629 WatchSource:0}: Error finding container 86f3f847aa31f15af7d74ad1a7719584b4fbc0ed4608b9adab02523677747629: Status 404 returned error can't find the container with id 86f3f847aa31f15af7d74ad1a7719584b4fbc0ed4608b9adab02523677747629
Sep 16 12:08:52.998887 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:52.998852    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76c88fdd8d-9dc4s" event={"ID":"28a3cb28-215a-41e1-86f9-bcc4c0923e7c","Type":"ContainerStarted","Data":"002e49d6e311a142435a4e6e05959f347371f901356fef4f40d853eb931d9371"}
Sep 16 12:08:52.999078 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:52.998899    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76c88fdd8d-9dc4s" event={"ID":"28a3cb28-215a-41e1-86f9-bcc4c0923e7c","Type":"ContainerStarted","Data":"86f3f847aa31f15af7d74ad1a7719584b4fbc0ed4608b9adab02523677747629"}
Sep 16 12:08:53.023406 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:08:53.023354    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76c88fdd8d-9dc4s" podStartSLOduration=2.023337425 podStartE2EDuration="2.023337425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:08:53.021881096 +0000 UTC m=+326.722952421" watchObservedRunningTime="2026-09-16 12:08:53.023337425 +0000 UTC m=+326.724408750"
Sep 16 12:09:01.687232 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:01.687185    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:09:01.687834 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:01.687254    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:09:01.691984 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:01.691963    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:09:02.041818 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:02.041746    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:09:02.546182 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:02.546144    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:09:02.546347 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:02.546214    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:09:03.042481 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:03.042446    2579 generic.go:356] "Generic (PLEG): container finished" podID="d7c6dfde-e151-4ca0-86bf-3cd8b766ee53" containerID="0b2d628e429583115dc2a7e3fd4747929532fa587e4284460a9e5247d25be259" exitCode=0
Sep 16 12:09:03.042897 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:03.042521    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xqnvz" event={"ID":"d7c6dfde-e151-4ca0-86bf-3cd8b766ee53","Type":"ContainerDied","Data":"0b2d628e429583115dc2a7e3fd4747929532fa587e4284460a9e5247d25be259"}
Sep 16 12:09:03.042955 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:03.042939    2579 scope.go:117] "RemoveContainer" containerID="0b2d628e429583115dc2a7e3fd4747929532fa587e4284460a9e5247d25be259"
Sep 16 12:09:04.047700 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:04.047649    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-xqnvz" event={"ID":"d7c6dfde-e151-4ca0-86bf-3cd8b766ee53","Type":"ContainerStarted","Data":"74d4c4078201b3a7e898326fc3f96dc84bd7f9994c4b69aaf5be1ed046d31588"}
Sep 16 12:09:13.999888 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:13.999820    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-76bd8984d-9md9w" podUID="ed5ab3f0-c343-47c0-9695-ebcbed31ec77" containerName="console" containerID="cri-o://608597662f6731a01c762e4aeb3b946b31a9773ccbb82fd2b4b0d9dcb713ffbb" gracePeriod=15
Sep 16 12:09:14.269966 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:14.269888    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:09:14.388916 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:14.388878    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-console-serving-cert\") pod \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") "
Sep 16 12:09:14.388916 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:14.388918    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-oauth-serving-cert\") pod \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") "
Sep 16 12:09:14.389165 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:14.388948    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-console-oauth-config\") pod \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") "
Sep 16 12:09:14.389165 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:14.389001    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-console-config\") pod \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") "
Sep 16 12:09:14.389165 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:14.389025    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spmnx\" (UniqueName: \"kubernetes.io/projected/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-kube-api-access-spmnx\") pod \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") "
Sep 16 12:09:14.389165 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:14.389048    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-trusted-ca-bundle\") pod \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") "
Sep 16 12:09:14.389376 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:14.389182    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-service-ca\") pod \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\" (UID: \"ed5ab3f0-c343-47c0-9695-ebcbed31ec77\") "
Sep 16 12:09:14.389436 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:14.389318    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ed5ab3f0-c343-47c0-9695-ebcbed31ec77" (UID: "ed5ab3f0-c343-47c0-9695-ebcbed31ec77"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:09:14.389568 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:14.389539    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ed5ab3f0-c343-47c0-9695-ebcbed31ec77" (UID: "ed5ab3f0-c343-47c0-9695-ebcbed31ec77"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:09:14.389718 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:14.389651    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-service-ca" (OuterVolumeSpecName: "service-ca") pod "ed5ab3f0-c343-47c0-9695-ebcbed31ec77" (UID: "ed5ab3f0-c343-47c0-9695-ebcbed31ec77"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:09:14.389718 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:14.389683    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-console-config" (OuterVolumeSpecName: "console-config") pod "ed5ab3f0-c343-47c0-9695-ebcbed31ec77" (UID: "ed5ab3f0-c343-47c0-9695-ebcbed31ec77"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:09:14.391263 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:14.391242    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ed5ab3f0-c343-47c0-9695-ebcbed31ec77" (UID: "ed5ab3f0-c343-47c0-9695-ebcbed31ec77"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:09:14.391364 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:14.391319    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-kube-api-access-spmnx" (OuterVolumeSpecName: "kube-api-access-spmnx") pod "ed5ab3f0-c343-47c0-9695-ebcbed31ec77" (UID: "ed5ab3f0-c343-47c0-9695-ebcbed31ec77"). InnerVolumeSpecName "kube-api-access-spmnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:09:14.391364 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:14.391345    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ed5ab3f0-c343-47c0-9695-ebcbed31ec77" (UID: "ed5ab3f0-c343-47c0-9695-ebcbed31ec77"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:09:14.490468 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:14.490428    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-console-serving-cert\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:09:14.490468 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:14.490459    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-oauth-serving-cert\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:09:14.490468 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:14.490468    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-console-oauth-config\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:09:14.490468 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:14.490477    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-console-config\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:09:14.490753 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:14.490486    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spmnx\" (UniqueName: \"kubernetes.io/projected/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-kube-api-access-spmnx\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:09:14.490753 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:14.490494    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-trusted-ca-bundle\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:09:14.490753 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:14.490502    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed5ab3f0-c343-47c0-9695-ebcbed31ec77-service-ca\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:09:15.092911 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:15.092880    2579 generic.go:356] "Generic (PLEG): container finished" podID="ed5ab3f0-c343-47c0-9695-ebcbed31ec77" containerID="608597662f6731a01c762e4aeb3b946b31a9773ccbb82fd2b4b0d9dcb713ffbb" exitCode=0
Sep 16 12:09:15.093304 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:15.092924    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76bd8984d-9md9w"
Sep 16 12:09:15.093304 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:15.092961    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76bd8984d-9md9w" event={"ID":"ed5ab3f0-c343-47c0-9695-ebcbed31ec77","Type":"ContainerDied","Data":"608597662f6731a01c762e4aeb3b946b31a9773ccbb82fd2b4b0d9dcb713ffbb"}
Sep 16 12:09:15.093304 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:15.093007    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76bd8984d-9md9w" event={"ID":"ed5ab3f0-c343-47c0-9695-ebcbed31ec77","Type":"ContainerDied","Data":"54f29443782e45f5e2c75b47be5ab962a5d732d061e8f1107181c273cd972edd"}
Sep 16 12:09:15.093304 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:15.093028    2579 scope.go:117] "RemoveContainer" containerID="608597662f6731a01c762e4aeb3b946b31a9773ccbb82fd2b4b0d9dcb713ffbb"
Sep 16 12:09:15.101264 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:15.101243    2579 scope.go:117] "RemoveContainer" containerID="608597662f6731a01c762e4aeb3b946b31a9773ccbb82fd2b4b0d9dcb713ffbb"
Sep 16 12:09:15.101530 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:09:15.101510    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"608597662f6731a01c762e4aeb3b946b31a9773ccbb82fd2b4b0d9dcb713ffbb\": container with ID starting with 608597662f6731a01c762e4aeb3b946b31a9773ccbb82fd2b4b0d9dcb713ffbb not found: ID does not exist" containerID="608597662f6731a01c762e4aeb3b946b31a9773ccbb82fd2b4b0d9dcb713ffbb"
Sep 16 12:09:15.101613 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:15.101539    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608597662f6731a01c762e4aeb3b946b31a9773ccbb82fd2b4b0d9dcb713ffbb"} err="failed to get container status \"608597662f6731a01c762e4aeb3b946b31a9773ccbb82fd2b4b0d9dcb713ffbb\": rpc error: code = NotFound desc = could not find container \"608597662f6731a01c762e4aeb3b946b31a9773ccbb82fd2b4b0d9dcb713ffbb\": container with ID starting with 608597662f6731a01c762e4aeb3b946b31a9773ccbb82fd2b4b0d9dcb713ffbb not found: ID does not exist"
Sep 16 12:09:15.108855 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:15.108835    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76bd8984d-9md9w"]
Sep 16 12:09:15.112068 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:15.112049    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76bd8984d-9md9w"]
Sep 16 12:09:16.778860 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:16.778828    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed5ab3f0-c343-47c0-9695-ebcbed31ec77" path="/var/lib/kubelet/pods/ed5ab3f0-c343-47c0-9695-ebcbed31ec77/volumes"
Sep 16 12:09:22.125871 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:22.125837    2579 generic.go:356] "Generic (PLEG): container finished" podID="f2dca0f8-99b0-4189-b1c0-329c54fdbc24" containerID="f6069bf683fa51d5f49b2e21c651675da928d796dd792c132f93ecc1b2f34b8b" exitCode=0
Sep 16 12:09:22.126420 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:22.125915    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-8f59b" event={"ID":"f2dca0f8-99b0-4189-b1c0-329c54fdbc24","Type":"ContainerDied","Data":"f6069bf683fa51d5f49b2e21c651675da928d796dd792c132f93ecc1b2f34b8b"}
Sep 16 12:09:22.126420 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:22.126260    2579 scope.go:117] "RemoveContainer" containerID="f6069bf683fa51d5f49b2e21c651675da928d796dd792c132f93ecc1b2f34b8b"
Sep 16 12:09:22.553321 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:22.553246    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:09:22.558332 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:22.558305    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-dc766c59d-hfznn"
Sep 16 12:09:23.131351 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:23.131319    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-8f59b" event={"ID":"f2dca0f8-99b0-4189-b1c0-329c54fdbc24","Type":"ContainerStarted","Data":"05db62e9c17f3f0cdeece7fe648b65a63898bfef1c0c924b7b46eb3f5b92e1d9"}
Sep 16 12:09:40.500057 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:40.500015    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs\") pod \"network-metrics-daemon-7cn45\" (UID: \"5441211d-70c6-473a-ac90-417bd7aa57ec\") " pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:09:40.504045 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:40.503987    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5441211d-70c6-473a-ac90-417bd7aa57ec-metrics-certs\") pod \"network-metrics-daemon-7cn45\" (UID: \"5441211d-70c6-473a-ac90-417bd7aa57ec\") " pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:09:40.680168 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:40.680139    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8kdnz\""
Sep 16 12:09:40.688648 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:40.688374    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7cn45"
Sep 16 12:09:40.980953 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:40.980911    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7cn45"]
Sep 16 12:09:40.983327 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:09:40.983298    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5441211d_70c6_473a_ac90_417bd7aa57ec.slice/crio-3c895bca80a4dded6adf2bd8de21274ad365e5c495465420dfc4906f6f5f537a WatchSource:0}: Error finding container 3c895bca80a4dded6adf2bd8de21274ad365e5c495465420dfc4906f6f5f537a: Status 404 returned error can't find the container with id 3c895bca80a4dded6adf2bd8de21274ad365e5c495465420dfc4906f6f5f537a
Sep 16 12:09:41.199446 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:41.199338    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7cn45" event={"ID":"5441211d-70c6-473a-ac90-417bd7aa57ec","Type":"ContainerStarted","Data":"3c895bca80a4dded6adf2bd8de21274ad365e5c495465420dfc4906f6f5f537a"}
Sep 16 12:09:42.207409 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:42.207324    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7cn45" event={"ID":"5441211d-70c6-473a-ac90-417bd7aa57ec","Type":"ContainerStarted","Data":"81e89406a11c0fe7f268b3ccb54f35a99f3f0362d82099dae813294e0f59a1b8"}
Sep 16 12:09:42.207409 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:42.207383    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7cn45" event={"ID":"5441211d-70c6-473a-ac90-417bd7aa57ec","Type":"ContainerStarted","Data":"8620c796d59d0d28df8be0cbdc970644527a855b257e4ce8582149ba234123a9"}
Sep 16 12:09:42.222518 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:42.222458    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7cn45" podStartSLOduration=374.279194523 podStartE2EDuration="6m15.222434379s" totalImagesPullingTime="943.239856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:09:42.22170684 +0000 UTC m=+375.922778169" watchObservedRunningTime="2026-09-16 12:09:42.222434379 +0000 UTC m=+375.923505705"
Sep 16 12:09:58.176260 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:58.176224    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:09:58.176739 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:58.176498    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="alertmanager" containerID="cri-o://baae7854507279469f63913b721421cea49f7fc8b4bd107af5ce3133630d4364" gracePeriod=120
Sep 16 12:09:58.176739 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:58.176566    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="kube-rbac-proxy-metric" containerID="cri-o://8c20c00f59afb8cc2cfabedfa6cad362c8c46d123ab0cc651bf024ab94d10e6a" gracePeriod=120
Sep 16 12:09:58.176739 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:58.176624    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="prom-label-proxy" containerID="cri-o://3a7c4ea08e4a2f89c1c2bf13c5109961d58546f72f686d85e3633ad4d3677365" gracePeriod=120
Sep 16 12:09:58.176918 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:58.176621    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="kube-rbac-proxy-web" containerID="cri-o://3248f430b8759cde9f4c31b0210825e560a785d13852c802856f8442b6e5d747" gracePeriod=120
Sep 16 12:09:58.176975 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:58.176677    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="kube-rbac-proxy" containerID="cri-o://8b496967ebd0664b497ac532fbe84103b74388a5c8e2ff520b283c7e6908acb7" gracePeriod=120
Sep 16 12:09:58.177464 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:58.176641    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="config-reloader" containerID="cri-o://b6cfef3f3b52d5141fe22f151542dba259dd3c9c3b44ff154a091d868656142b" gracePeriod=120
Sep 16 12:09:59.277017 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.276991    2579 generic.go:356] "Generic (PLEG): container finished" podID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerID="3a7c4ea08e4a2f89c1c2bf13c5109961d58546f72f686d85e3633ad4d3677365" exitCode=0
Sep 16 12:09:59.277017 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.277014    2579 generic.go:356] "Generic (PLEG): container finished" podID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerID="8c20c00f59afb8cc2cfabedfa6cad362c8c46d123ab0cc651bf024ab94d10e6a" exitCode=0
Sep 16 12:09:59.277402 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.277023    2579 generic.go:356] "Generic (PLEG): container finished" podID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerID="8b496967ebd0664b497ac532fbe84103b74388a5c8e2ff520b283c7e6908acb7" exitCode=0
Sep 16 12:09:59.277402 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.277028    2579 generic.go:356] "Generic (PLEG): container finished" podID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerID="b6cfef3f3b52d5141fe22f151542dba259dd3c9c3b44ff154a091d868656142b" exitCode=0
Sep 16 12:09:59.277402 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.277034    2579 generic.go:356] "Generic (PLEG): container finished" podID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerID="baae7854507279469f63913b721421cea49f7fc8b4bd107af5ce3133630d4364" exitCode=0
Sep 16 12:09:59.277402 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.277064    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9dc098ac-4cd7-47c2-a026-e6e933e20f4f","Type":"ContainerDied","Data":"3a7c4ea08e4a2f89c1c2bf13c5109961d58546f72f686d85e3633ad4d3677365"}
Sep 16 12:09:59.277402 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.277151    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9dc098ac-4cd7-47c2-a026-e6e933e20f4f","Type":"ContainerDied","Data":"8c20c00f59afb8cc2cfabedfa6cad362c8c46d123ab0cc651bf024ab94d10e6a"}
Sep 16 12:09:59.277402 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.277206    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9dc098ac-4cd7-47c2-a026-e6e933e20f4f","Type":"ContainerDied","Data":"8b496967ebd0664b497ac532fbe84103b74388a5c8e2ff520b283c7e6908acb7"}
Sep 16 12:09:59.277402 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.277249    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9dc098ac-4cd7-47c2-a026-e6e933e20f4f","Type":"ContainerDied","Data":"b6cfef3f3b52d5141fe22f151542dba259dd3c9c3b44ff154a091d868656142b"}
Sep 16 12:09:59.277402 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.277289    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9dc098ac-4cd7-47c2-a026-e6e933e20f4f","Type":"ContainerDied","Data":"baae7854507279469f63913b721421cea49f7fc8b4bd107af5ce3133630d4364"}
Sep 16 12:09:59.414554 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.414532    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:09:59.564604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.564518    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-web-config\") pod \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") "
Sep 16 12:09:59.564604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.564565    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9jvh\" (UniqueName: \"kubernetes.io/projected/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-kube-api-access-z9jvh\") pod \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") "
Sep 16 12:09:59.564604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.564583    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-main-tls\") pod \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") "
Sep 16 12:09:59.564907 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.564612    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-config-volume\") pod \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") "
Sep 16 12:09:59.564907 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.564631    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-kube-rbac-proxy-web\") pod \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") "
Sep 16 12:09:59.564907 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.564649    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-cluster-tls-config\") pod \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") "
Sep 16 12:09:59.564907 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.564696    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-alertmanager-main-db\") pod \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") "
Sep 16 12:09:59.565111 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.565019    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "9dc098ac-4cd7-47c2-a026-e6e933e20f4f" (UID: "9dc098ac-4cd7-47c2-a026-e6e933e20f4f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:09:59.565176 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.565103    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-kube-rbac-proxy\") pod \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") "
Sep 16 12:09:59.565176 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.565152    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") "
Sep 16 12:09:59.565276 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.565190    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-metrics-client-ca\") pod \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") "
Sep 16 12:09:59.565697 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.565439    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-config-out\") pod \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") "
Sep 16 12:09:59.565697 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.565491    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-tls-assets\") pod \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") "
Sep 16 12:09:59.565697 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.565492    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "9dc098ac-4cd7-47c2-a026-e6e933e20f4f" (UID: "9dc098ac-4cd7-47c2-a026-e6e933e20f4f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:09:59.565697 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.565551    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-alertmanager-trusted-ca-bundle\") pod \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\" (UID: \"9dc098ac-4cd7-47c2-a026-e6e933e20f4f\") "
Sep 16 12:09:59.566052 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.565805    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-alertmanager-main-db\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:09:59.566052 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.565824    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-metrics-client-ca\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:09:59.566154 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.566113    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "9dc098ac-4cd7-47c2-a026-e6e933e20f4f" (UID: "9dc098ac-4cd7-47c2-a026-e6e933e20f4f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:09:59.567339 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.567303    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "9dc098ac-4cd7-47c2-a026-e6e933e20f4f" (UID: "9dc098ac-4cd7-47c2-a026-e6e933e20f4f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:09:59.567584 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.567548    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-kube-api-access-z9jvh" (OuterVolumeSpecName: "kube-api-access-z9jvh") pod "9dc098ac-4cd7-47c2-a026-e6e933e20f4f" (UID: "9dc098ac-4cd7-47c2-a026-e6e933e20f4f"). InnerVolumeSpecName "kube-api-access-z9jvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:09:59.568176 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.568146    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-config-out" (OuterVolumeSpecName: "config-out") pod "9dc098ac-4cd7-47c2-a026-e6e933e20f4f" (UID: "9dc098ac-4cd7-47c2-a026-e6e933e20f4f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:09:59.568176 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.568160    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-config-volume" (OuterVolumeSpecName: "config-volume") pod "9dc098ac-4cd7-47c2-a026-e6e933e20f4f" (UID: "9dc098ac-4cd7-47c2-a026-e6e933e20f4f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:09:59.568341 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.568321    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "9dc098ac-4cd7-47c2-a026-e6e933e20f4f" (UID: "9dc098ac-4cd7-47c2-a026-e6e933e20f4f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:09:59.568767 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.568727    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "9dc098ac-4cd7-47c2-a026-e6e933e20f4f" (UID: "9dc098ac-4cd7-47c2-a026-e6e933e20f4f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:09:59.568900 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.568882    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "9dc098ac-4cd7-47c2-a026-e6e933e20f4f" (UID: "9dc098ac-4cd7-47c2-a026-e6e933e20f4f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:09:59.569045 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.569031    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "9dc098ac-4cd7-47c2-a026-e6e933e20f4f" (UID: "9dc098ac-4cd7-47c2-a026-e6e933e20f4f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:09:59.571898 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.571876    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "9dc098ac-4cd7-47c2-a026-e6e933e20f4f" (UID: "9dc098ac-4cd7-47c2-a026-e6e933e20f4f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:09:59.577997 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.577976    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-web-config" (OuterVolumeSpecName: "web-config") pod "9dc098ac-4cd7-47c2-a026-e6e933e20f4f" (UID: "9dc098ac-4cd7-47c2-a026-e6e933e20f4f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:09:59.666948 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.666914    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:09:59.666948 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.666941    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-web-config\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:09:59.666948 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.666951    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9jvh\" (UniqueName: \"kubernetes.io/projected/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-kube-api-access-z9jvh\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:09:59.667157 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.666960    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-main-tls\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:09:59.667157 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.666969    2579 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-config-volume\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:09:59.667157 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.666977    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:09:59.667157 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.666985    2579 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-cluster-tls-config\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:09:59.667157 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.666994    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:09:59.667157 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.667002    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:09:59.667157 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.667011    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-config-out\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:09:59.667157 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:09:59.667018    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9dc098ac-4cd7-47c2-a026-e6e933e20f4f-tls-assets\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:10:00.283791 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.283759    2579 generic.go:356] "Generic (PLEG): container finished" podID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerID="3248f430b8759cde9f4c31b0210825e560a785d13852c802856f8442b6e5d747" exitCode=0
Sep 16 12:10:00.284120 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.283849    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.284120 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.283854    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9dc098ac-4cd7-47c2-a026-e6e933e20f4f","Type":"ContainerDied","Data":"3248f430b8759cde9f4c31b0210825e560a785d13852c802856f8442b6e5d747"}
Sep 16 12:10:00.284120 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.283930    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9dc098ac-4cd7-47c2-a026-e6e933e20f4f","Type":"ContainerDied","Data":"a9ac5f6aaec705e744bbd140865d7f8bd8a60e003f12ad9d7a973d416d0cb951"}
Sep 16 12:10:00.284120 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.283992    2579 scope.go:117] "RemoveContainer" containerID="3a7c4ea08e4a2f89c1c2bf13c5109961d58546f72f686d85e3633ad4d3677365"
Sep 16 12:10:00.292455 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.292329    2579 scope.go:117] "RemoveContainer" containerID="8c20c00f59afb8cc2cfabedfa6cad362c8c46d123ab0cc651bf024ab94d10e6a"
Sep 16 12:10:00.299541 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.299522    2579 scope.go:117] "RemoveContainer" containerID="8b496967ebd0664b497ac532fbe84103b74388a5c8e2ff520b283c7e6908acb7"
Sep 16 12:10:00.306833 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.306808    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:10:00.307585 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.307571    2579 scope.go:117] "RemoveContainer" containerID="3248f430b8759cde9f4c31b0210825e560a785d13852c802856f8442b6e5d747"
Sep 16 12:10:00.315898 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.315866    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:10:00.318645 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.318625    2579 scope.go:117] "RemoveContainer" containerID="b6cfef3f3b52d5141fe22f151542dba259dd3c9c3b44ff154a091d868656142b"
Sep 16 12:10:00.326090 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.326071    2579 scope.go:117] "RemoveContainer" containerID="baae7854507279469f63913b721421cea49f7fc8b4bd107af5ce3133630d4364"
Sep 16 12:10:00.333728 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.333640    2579 scope.go:117] "RemoveContainer" containerID="2c71954d27540bdd08c0af9995784272353b79004822d87bd7154f3a9a3f4aba"
Sep 16 12:10:00.336738 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.336713    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:10:00.337037 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.337000    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="alertmanager"
Sep 16 12:10:00.337037 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.337038    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="alertmanager"
Sep 16 12:10:00.337157 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.337048    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="kube-rbac-proxy-metric"
Sep 16 12:10:00.337157 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.337054    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="kube-rbac-proxy-metric"
Sep 16 12:10:00.337157 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.337063    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed5ab3f0-c343-47c0-9695-ebcbed31ec77" containerName="console"
Sep 16 12:10:00.337157 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.337068    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed5ab3f0-c343-47c0-9695-ebcbed31ec77" containerName="console"
Sep 16 12:10:00.337157 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.337079    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="prom-label-proxy"
Sep 16 12:10:00.337157 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.337084    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="prom-label-proxy"
Sep 16 12:10:00.337157 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.337092    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="kube-rbac-proxy-web"
Sep 16 12:10:00.337157 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.337097    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="kube-rbac-proxy-web"
Sep 16 12:10:00.337157 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.337103    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="kube-rbac-proxy"
Sep 16 12:10:00.337157 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.337108    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="kube-rbac-proxy"
Sep 16 12:10:00.337157 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.337116    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="init-config-reloader"
Sep 16 12:10:00.337157 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.337121    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="init-config-reloader"
Sep 16 12:10:00.337157 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.337126    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="config-reloader"
Sep 16 12:10:00.337157 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.337131    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="config-reloader"
Sep 16 12:10:00.337621 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.337186    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed5ab3f0-c343-47c0-9695-ebcbed31ec77" containerName="console"
Sep 16 12:10:00.337621 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.337194    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="prom-label-proxy"
Sep 16 12:10:00.337621 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.337201    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="config-reloader"
Sep 16 12:10:00.337621 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.337206    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="kube-rbac-proxy"
Sep 16 12:10:00.337621 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.337212    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="kube-rbac-proxy-web"
Sep 16 12:10:00.337621 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.337216    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="kube-rbac-proxy-metric"
Sep 16 12:10:00.337621 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.337222    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" containerName="alertmanager"
Sep 16 12:10:00.340898 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.340881    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.341784 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.341764    2579 scope.go:117] "RemoveContainer" containerID="3a7c4ea08e4a2f89c1c2bf13c5109961d58546f72f686d85e3633ad4d3677365"
Sep 16 12:10:00.342047 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:10:00.342027    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a7c4ea08e4a2f89c1c2bf13c5109961d58546f72f686d85e3633ad4d3677365\": container with ID starting with 3a7c4ea08e4a2f89c1c2bf13c5109961d58546f72f686d85e3633ad4d3677365 not found: ID does not exist" containerID="3a7c4ea08e4a2f89c1c2bf13c5109961d58546f72f686d85e3633ad4d3677365"
Sep 16 12:10:00.342104 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.342055    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a7c4ea08e4a2f89c1c2bf13c5109961d58546f72f686d85e3633ad4d3677365"} err="failed to get container status \"3a7c4ea08e4a2f89c1c2bf13c5109961d58546f72f686d85e3633ad4d3677365\": rpc error: code = NotFound desc = could not find container \"3a7c4ea08e4a2f89c1c2bf13c5109961d58546f72f686d85e3633ad4d3677365\": container with ID starting with 3a7c4ea08e4a2f89c1c2bf13c5109961d58546f72f686d85e3633ad4d3677365 not found: ID does not exist"
Sep 16 12:10:00.342104 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.342078    2579 scope.go:117] "RemoveContainer" containerID="8c20c00f59afb8cc2cfabedfa6cad362c8c46d123ab0cc651bf024ab94d10e6a"
Sep 16 12:10:00.342351 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:10:00.342335    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c20c00f59afb8cc2cfabedfa6cad362c8c46d123ab0cc651bf024ab94d10e6a\": container with ID starting with 8c20c00f59afb8cc2cfabedfa6cad362c8c46d123ab0cc651bf024ab94d10e6a not found: ID does not exist" containerID="8c20c00f59afb8cc2cfabedfa6cad362c8c46d123ab0cc651bf024ab94d10e6a"
Sep 16 12:10:00.342398 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.342355    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c20c00f59afb8cc2cfabedfa6cad362c8c46d123ab0cc651bf024ab94d10e6a"} err="failed to get container status \"8c20c00f59afb8cc2cfabedfa6cad362c8c46d123ab0cc651bf024ab94d10e6a\": rpc error: code = NotFound desc = could not find container \"8c20c00f59afb8cc2cfabedfa6cad362c8c46d123ab0cc651bf024ab94d10e6a\": container with ID starting with 8c20c00f59afb8cc2cfabedfa6cad362c8c46d123ab0cc651bf024ab94d10e6a not found: ID does not exist"
Sep 16 12:10:00.342398 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.342369    2579 scope.go:117] "RemoveContainer" containerID="8b496967ebd0664b497ac532fbe84103b74388a5c8e2ff520b283c7e6908acb7"
Sep 16 12:10:00.342571 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:10:00.342554    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b496967ebd0664b497ac532fbe84103b74388a5c8e2ff520b283c7e6908acb7\": container with ID starting with 8b496967ebd0664b497ac532fbe84103b74388a5c8e2ff520b283c7e6908acb7 not found: ID does not exist" containerID="8b496967ebd0664b497ac532fbe84103b74388a5c8e2ff520b283c7e6908acb7"
Sep 16 12:10:00.342616 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.342573    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b496967ebd0664b497ac532fbe84103b74388a5c8e2ff520b283c7e6908acb7"} err="failed to get container status \"8b496967ebd0664b497ac532fbe84103b74388a5c8e2ff520b283c7e6908acb7\": rpc error: code = NotFound desc = could not find container \"8b496967ebd0664b497ac532fbe84103b74388a5c8e2ff520b283c7e6908acb7\": container with ID starting with 8b496967ebd0664b497ac532fbe84103b74388a5c8e2ff520b283c7e6908acb7 not found: ID does not exist"
Sep 16 12:10:00.342616 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.342585    2579 scope.go:117] "RemoveContainer" containerID="3248f430b8759cde9f4c31b0210825e560a785d13852c802856f8442b6e5d747"
Sep 16 12:10:00.342867 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:10:00.342851    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3248f430b8759cde9f4c31b0210825e560a785d13852c802856f8442b6e5d747\": container with ID starting with 3248f430b8759cde9f4c31b0210825e560a785d13852c802856f8442b6e5d747 not found: ID does not exist" containerID="3248f430b8759cde9f4c31b0210825e560a785d13852c802856f8442b6e5d747"
Sep 16 12:10:00.342923 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.342876    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3248f430b8759cde9f4c31b0210825e560a785d13852c802856f8442b6e5d747"} err="failed to get container status \"3248f430b8759cde9f4c31b0210825e560a785d13852c802856f8442b6e5d747\": rpc error: code = NotFound desc = could not find container \"3248f430b8759cde9f4c31b0210825e560a785d13852c802856f8442b6e5d747\": container with ID starting with 3248f430b8759cde9f4c31b0210825e560a785d13852c802856f8442b6e5d747 not found: ID does not exist"
Sep 16 12:10:00.342923 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.342898    2579 scope.go:117] "RemoveContainer" containerID="b6cfef3f3b52d5141fe22f151542dba259dd3c9c3b44ff154a091d868656142b"
Sep 16 12:10:00.343152 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:10:00.343133    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6cfef3f3b52d5141fe22f151542dba259dd3c9c3b44ff154a091d868656142b\": container with ID starting with b6cfef3f3b52d5141fe22f151542dba259dd3c9c3b44ff154a091d868656142b not found: ID does not exist" containerID="b6cfef3f3b52d5141fe22f151542dba259dd3c9c3b44ff154a091d868656142b"
Sep 16 12:10:00.343223 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.343168    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6cfef3f3b52d5141fe22f151542dba259dd3c9c3b44ff154a091d868656142b"} err="failed to get container status \"b6cfef3f3b52d5141fe22f151542dba259dd3c9c3b44ff154a091d868656142b\": rpc error: code = NotFound desc = could not find container \"b6cfef3f3b52d5141fe22f151542dba259dd3c9c3b44ff154a091d868656142b\": container with ID starting with b6cfef3f3b52d5141fe22f151542dba259dd3c9c3b44ff154a091d868656142b not found: ID does not exist"
Sep 16 12:10:00.343223 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.343189    2579 scope.go:117] "RemoveContainer" containerID="baae7854507279469f63913b721421cea49f7fc8b4bd107af5ce3133630d4364"
Sep 16 12:10:00.343619 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.343350    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 12:10:00.343619 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.343405    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 12:10:00.343619 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.343437    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 12:10:00.343619 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.343481    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 12:10:00.343619 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.343492    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 12:10:00.343619 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.343408    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 12:10:00.343619 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:10:00.343597    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baae7854507279469f63913b721421cea49f7fc8b4bd107af5ce3133630d4364\": container with ID starting with baae7854507279469f63913b721421cea49f7fc8b4bd107af5ce3133630d4364 not found: ID does not exist" containerID="baae7854507279469f63913b721421cea49f7fc8b4bd107af5ce3133630d4364"
Sep 16 12:10:00.343619 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.343621    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baae7854507279469f63913b721421cea49f7fc8b4bd107af5ce3133630d4364"} err="failed to get container status \"baae7854507279469f63913b721421cea49f7fc8b4bd107af5ce3133630d4364\": rpc error: code = NotFound desc = could not find container \"baae7854507279469f63913b721421cea49f7fc8b4bd107af5ce3133630d4364\": container with ID starting with baae7854507279469f63913b721421cea49f7fc8b4bd107af5ce3133630d4364 not found: ID does not exist"
Sep 16 12:10:00.343619 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.343640    2579 scope.go:117] "RemoveContainer" containerID="2c71954d27540bdd08c0af9995784272353b79004822d87bd7154f3a9a3f4aba"
Sep 16 12:10:00.344185 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.343718    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7cwd5\""
Sep 16 12:10:00.344185 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.343825    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 12:10:00.344185 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.343911    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 12:10:00.344185 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:10:00.343941    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c71954d27540bdd08c0af9995784272353b79004822d87bd7154f3a9a3f4aba\": container with ID starting with 2c71954d27540bdd08c0af9995784272353b79004822d87bd7154f3a9a3f4aba not found: ID does not exist" containerID="2c71954d27540bdd08c0af9995784272353b79004822d87bd7154f3a9a3f4aba"
Sep 16 12:10:00.344185 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.343966    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c71954d27540bdd08c0af9995784272353b79004822d87bd7154f3a9a3f4aba"} err="failed to get container status \"2c71954d27540bdd08c0af9995784272353b79004822d87bd7154f3a9a3f4aba\": rpc error: code = NotFound desc = could not find container \"2c71954d27540bdd08c0af9995784272353b79004822d87bd7154f3a9a3f4aba\": container with ID starting with 2c71954d27540bdd08c0af9995784272353b79004822d87bd7154f3a9a3f4aba not found: ID does not exist"
Sep 16 12:10:00.348859 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.348840    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 12:10:00.356925 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.356905    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:10:00.472746 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.472696    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-config-out\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.472746 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.472755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.472962 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.472819    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.472962 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.472865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.472962 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.472882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.472962 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.472898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.472962 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.472920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.472962 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.472934    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.472962 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.472951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-config-volume\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.472962 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.472965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmml5\" (UniqueName: \"kubernetes.io/projected/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-kube-api-access-zmml5\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.473219 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.472987    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.473219 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.473028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-web-config\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.473219 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.473066    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.574453 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.574369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.574453 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.574423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.574453 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.574457    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.574674 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.574481    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.574674 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.574506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.574674 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.574534    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.574674 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.574556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.574674 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.574584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-config-volume\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.574674 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.574603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmml5\" (UniqueName: \"kubernetes.io/projected/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-kube-api-access-zmml5\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.574674 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.574646    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.575002 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.574686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-web-config\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.575002 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.574714    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.575002 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.574750    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-config-out\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.575002 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.574992    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.575797 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.575393    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.576955 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.576307    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.577676 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.577639    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-config-volume\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.577777 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.577724    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-config-out\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.577777 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.577750    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.578009 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.577980    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.578215 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.578197    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.578294 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.578233    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.578455 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.578429    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.579123 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.579099    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-web-config\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.579795 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.579777    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.584634 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.584614    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmml5\" (UniqueName: \"kubernetes.io/projected/fbf3df07-a75e-413d-8f60-ed4bd2bb9e08-kube-api-access-zmml5\") pod \"alertmanager-main-0\" (UID: \"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.650288 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.650256    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 12:10:00.779931 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.779896    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dc098ac-4cd7-47c2-a026-e6e933e20f4f" path="/var/lib/kubelet/pods/9dc098ac-4cd7-47c2-a026-e6e933e20f4f/volumes"
Sep 16 12:10:00.929927 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:00.929902    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 12:10:00.931178 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:10:00.931153    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf3df07_a75e_413d_8f60_ed4bd2bb9e08.slice/crio-d07b9b25c998e893152334df96d968078fee8825f150a195b99bc43a85af2d9c WatchSource:0}: Error finding container d07b9b25c998e893152334df96d968078fee8825f150a195b99bc43a85af2d9c: Status 404 returned error can't find the container with id d07b9b25c998e893152334df96d968078fee8825f150a195b99bc43a85af2d9c
Sep 16 12:10:01.290262 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:01.290178    2579 generic.go:356] "Generic (PLEG): container finished" podID="fbf3df07-a75e-413d-8f60-ed4bd2bb9e08" containerID="b4bc1a50ddb69442de4f40aefd70296821419e6d3f56df90b43182f8efe08c68" exitCode=0
Sep 16 12:10:01.290262 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:01.290214    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08","Type":"ContainerDied","Data":"b4bc1a50ddb69442de4f40aefd70296821419e6d3f56df90b43182f8efe08c68"}
Sep 16 12:10:01.290631 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:01.290272    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08","Type":"ContainerStarted","Data":"d07b9b25c998e893152334df96d968078fee8825f150a195b99bc43a85af2d9c"}
Sep 16 12:10:02.296704 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:02.296668    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08","Type":"ContainerStarted","Data":"91db22b9d756457ca7a76a26be86508226ebb348de0e367594bc979a5f5c83fd"}
Sep 16 12:10:02.297067 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:02.296735    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08","Type":"ContainerStarted","Data":"5cebb98728b00212b0f2a9849c56408f03b7b2eb8fd0a7c3fdcd3e5bdaf28b82"}
Sep 16 12:10:02.297067 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:02.296760    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08","Type":"ContainerStarted","Data":"7b145fed53068ebbfce25cacb63567d9e00447fd45ac6d7a8d7153d9a0b75ca2"}
Sep 16 12:10:02.297067 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:02.296785    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08","Type":"ContainerStarted","Data":"52a07b0be4e48ca78c4e11eb16e0bbdbd96c66992625609703e36c736ba2e167"}
Sep 16 12:10:02.297067 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:02.296809    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08","Type":"ContainerStarted","Data":"444560536adef1026a43dc9522090344619368d23402d22f583861c5e87eab01"}
Sep 16 12:10:02.297067 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:02.296831    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fbf3df07-a75e-413d-8f60-ed4bd2bb9e08","Type":"ContainerStarted","Data":"7041430ead1afd59682c1479d122f9b7006cb2d92489fb348f9d3ef091f1df23"}
Sep 16 12:10:02.322183 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:02.322127    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.305543631 podStartE2EDuration="2.322105338s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.561707ms" isStatefulPod=false podCreationTimestamp="2026-09-16 12:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:10:02.320556007 +0000 UTC m=+396.021627347" watchObservedRunningTime="2026-09-16 12:10:02.322105338 +0000 UTC m=+396.023176663"
Sep 16 12:10:12.207995 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:12.207960    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76c88fdd8d-9dc4s"]
Sep 16 12:10:37.226865 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:37.226823    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-76c88fdd8d-9dc4s" podUID="28a3cb28-215a-41e1-86f9-bcc4c0923e7c" containerName="console" containerID="cri-o://002e49d6e311a142435a4e6e05959f347371f901356fef4f40d853eb931d9371" gracePeriod=15
Sep 16 12:10:37.437203 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:37.437166    2579 generic.go:356] "Generic (PLEG): container finished" podID="28a3cb28-215a-41e1-86f9-bcc4c0923e7c" containerID="002e49d6e311a142435a4e6e05959f347371f901356fef4f40d853eb931d9371" exitCode=0
Sep 16 12:10:37.437349 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:37.437219    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76c88fdd8d-9dc4s" event={"ID":"28a3cb28-215a-41e1-86f9-bcc4c0923e7c","Type":"ContainerDied","Data":"002e49d6e311a142435a4e6e05959f347371f901356fef4f40d853eb931d9371"}
Sep 16 12:10:37.467664 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:37.467628    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:10:37.558054 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:37.557977    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-service-ca\") pod \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") "
Sep 16 12:10:37.558054 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:37.558019    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-trusted-ca-bundle\") pod \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") "
Sep 16 12:10:37.558248 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:37.558055    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glgrn\" (UniqueName: \"kubernetes.io/projected/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-kube-api-access-glgrn\") pod \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") "
Sep 16 12:10:37.558248 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:37.558165    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-console-serving-cert\") pod \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") "
Sep 16 12:10:37.558248 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:37.558211    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-console-oauth-config\") pod \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") "
Sep 16 12:10:37.558248 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:37.558233    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-oauth-serving-cert\") pod \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") "
Sep 16 12:10:37.558461 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:37.558257    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-console-config\") pod \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\" (UID: \"28a3cb28-215a-41e1-86f9-bcc4c0923e7c\") "
Sep 16 12:10:37.558545 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:37.558519    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-service-ca" (OuterVolumeSpecName: "service-ca") pod "28a3cb28-215a-41e1-86f9-bcc4c0923e7c" (UID: "28a3cb28-215a-41e1-86f9-bcc4c0923e7c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:10:37.558714 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:37.558598    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "28a3cb28-215a-41e1-86f9-bcc4c0923e7c" (UID: "28a3cb28-215a-41e1-86f9-bcc4c0923e7c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:10:37.558819 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:37.558760    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-console-config" (OuterVolumeSpecName: "console-config") pod "28a3cb28-215a-41e1-86f9-bcc4c0923e7c" (UID: "28a3cb28-215a-41e1-86f9-bcc4c0923e7c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:10:37.558819 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:37.558525    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "28a3cb28-215a-41e1-86f9-bcc4c0923e7c" (UID: "28a3cb28-215a-41e1-86f9-bcc4c0923e7c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:10:37.560197 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:37.560170    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "28a3cb28-215a-41e1-86f9-bcc4c0923e7c" (UID: "28a3cb28-215a-41e1-86f9-bcc4c0923e7c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:10:37.560290 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:37.560210    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "28a3cb28-215a-41e1-86f9-bcc4c0923e7c" (UID: "28a3cb28-215a-41e1-86f9-bcc4c0923e7c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:10:37.560290 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:37.560231    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-kube-api-access-glgrn" (OuterVolumeSpecName: "kube-api-access-glgrn") pod "28a3cb28-215a-41e1-86f9-bcc4c0923e7c" (UID: "28a3cb28-215a-41e1-86f9-bcc4c0923e7c"). InnerVolumeSpecName "kube-api-access-glgrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:10:37.659193 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:37.659159    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-console-oauth-config\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:10:37.659193 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:37.659185    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-oauth-serving-cert\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:10:37.659193 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:37.659194    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-console-config\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:10:37.659193 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:37.659202    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-service-ca\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:10:37.659434 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:37.659210    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-trusted-ca-bundle\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:10:37.659434 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:37.659219    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glgrn\" (UniqueName: \"kubernetes.io/projected/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-kube-api-access-glgrn\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:10:37.659434 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:37.659227    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/28a3cb28-215a-41e1-86f9-bcc4c0923e7c-console-serving-cert\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:10:38.442463 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:38.442431    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76c88fdd8d-9dc4s"
Sep 16 12:10:38.442943 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:38.442433    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76c88fdd8d-9dc4s" event={"ID":"28a3cb28-215a-41e1-86f9-bcc4c0923e7c","Type":"ContainerDied","Data":"86f3f847aa31f15af7d74ad1a7719584b4fbc0ed4608b9adab02523677747629"}
Sep 16 12:10:38.442943 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:38.442600    2579 scope.go:117] "RemoveContainer" containerID="002e49d6e311a142435a4e6e05959f347371f901356fef4f40d853eb931d9371"
Sep 16 12:10:38.462790 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:38.462766    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76c88fdd8d-9dc4s"]
Sep 16 12:10:38.467766 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:38.467741    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76c88fdd8d-9dc4s"]
Sep 16 12:10:38.779018 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:10:38.778945    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28a3cb28-215a-41e1-86f9-bcc4c0923e7c" path="/var/lib/kubelet/pods/28a3cb28-215a-41e1-86f9-bcc4c0923e7c/volumes"
Sep 16 12:11:08.682363 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:08.682323    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc"]
Sep 16 12:11:08.682861 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:08.682649    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28a3cb28-215a-41e1-86f9-bcc4c0923e7c" containerName="console"
Sep 16 12:11:08.682861 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:08.682677    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="28a3cb28-215a-41e1-86f9-bcc4c0923e7c" containerName="console"
Sep 16 12:11:08.682861 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:08.682742    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="28a3cb28-215a-41e1-86f9-bcc4c0923e7c" containerName="console"
Sep 16 12:11:08.686044 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:08.686027    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc"
Sep 16 12:11:08.688461 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:08.688438    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 12:11:08.688650 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:08.688628    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 12:11:08.689519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:08.689506    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r4zt6\""
Sep 16 12:11:08.693883 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:08.693860    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc"]
Sep 16 12:11:08.792394 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:08.792362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e5057ea-cc3e-46d4-8223-2c7334b9cc69-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc\" (UID: \"9e5057ea-cc3e-46d4-8223-2c7334b9cc69\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc"
Sep 16 12:11:08.792547 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:08.792417    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlfsx\" (UniqueName: \"kubernetes.io/projected/9e5057ea-cc3e-46d4-8223-2c7334b9cc69-kube-api-access-mlfsx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc\" (UID: \"9e5057ea-cc3e-46d4-8223-2c7334b9cc69\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc"
Sep 16 12:11:08.792547 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:08.792491    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e5057ea-cc3e-46d4-8223-2c7334b9cc69-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc\" (UID: \"9e5057ea-cc3e-46d4-8223-2c7334b9cc69\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc"
Sep 16 12:11:08.893562 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:08.893524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e5057ea-cc3e-46d4-8223-2c7334b9cc69-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc\" (UID: \"9e5057ea-cc3e-46d4-8223-2c7334b9cc69\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc"
Sep 16 12:11:08.893562 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:08.893564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e5057ea-cc3e-46d4-8223-2c7334b9cc69-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc\" (UID: \"9e5057ea-cc3e-46d4-8223-2c7334b9cc69\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc"
Sep 16 12:11:08.893794 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:08.893626    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlfsx\" (UniqueName: \"kubernetes.io/projected/9e5057ea-cc3e-46d4-8223-2c7334b9cc69-kube-api-access-mlfsx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc\" (UID: \"9e5057ea-cc3e-46d4-8223-2c7334b9cc69\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc"
Sep 16 12:11:08.893984 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:08.893962    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e5057ea-cc3e-46d4-8223-2c7334b9cc69-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc\" (UID: \"9e5057ea-cc3e-46d4-8223-2c7334b9cc69\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc"
Sep 16 12:11:08.894021 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:08.893972    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e5057ea-cc3e-46d4-8223-2c7334b9cc69-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc\" (UID: \"9e5057ea-cc3e-46d4-8223-2c7334b9cc69\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc"
Sep 16 12:11:08.901466 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:08.901443    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlfsx\" (UniqueName: \"kubernetes.io/projected/9e5057ea-cc3e-46d4-8223-2c7334b9cc69-kube-api-access-mlfsx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc\" (UID: \"9e5057ea-cc3e-46d4-8223-2c7334b9cc69\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc"
Sep 16 12:11:08.995309 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:08.995233    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc"
Sep 16 12:11:09.262077 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:09.262055    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc"]
Sep 16 12:11:09.264226 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:11:09.264191    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5057ea_cc3e_46d4_8223_2c7334b9cc69.slice/crio-aea6ef236691af1f47591fbd94dcc2bf34c57bcc9289a52336633c31e8ecdcc3 WatchSource:0}: Error finding container aea6ef236691af1f47591fbd94dcc2bf34c57bcc9289a52336633c31e8ecdcc3: Status 404 returned error can't find the container with id aea6ef236691af1f47591fbd94dcc2bf34c57bcc9289a52336633c31e8ecdcc3
Sep 16 12:11:09.560381 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:09.560306    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc" event={"ID":"9e5057ea-cc3e-46d4-8223-2c7334b9cc69","Type":"ContainerStarted","Data":"aea6ef236691af1f47591fbd94dcc2bf34c57bcc9289a52336633c31e8ecdcc3"}
Sep 16 12:11:16.593488 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:16.593456    2579 generic.go:356] "Generic (PLEG): container finished" podID="9e5057ea-cc3e-46d4-8223-2c7334b9cc69" containerID="5d999e25bbeb85e5adb687512a215e707a144bec64c86e965f1dcd3d13f3d269" exitCode=0
Sep 16 12:11:16.593867 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:16.593514    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc" event={"ID":"9e5057ea-cc3e-46d4-8223-2c7334b9cc69","Type":"ContainerDied","Data":"5d999e25bbeb85e5adb687512a215e707a144bec64c86e965f1dcd3d13f3d269"}
Sep 16 12:11:19.608455 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:19.608413    2579 generic.go:356] "Generic (PLEG): container finished" podID="9e5057ea-cc3e-46d4-8223-2c7334b9cc69" containerID="2904077673e3bfffc64e0162f03a72f8e01a388a6a327186bdd53ccba38482a0" exitCode=0
Sep 16 12:11:19.608855 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:19.608501    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc" event={"ID":"9e5057ea-cc3e-46d4-8223-2c7334b9cc69","Type":"ContainerDied","Data":"2904077673e3bfffc64e0162f03a72f8e01a388a6a327186bdd53ccba38482a0"}
Sep 16 12:11:26.639667 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:26.639619    2579 generic.go:356] "Generic (PLEG): container finished" podID="9e5057ea-cc3e-46d4-8223-2c7334b9cc69" containerID="3314537bf51d439d0893b4af7405061d520575cb85288bfb0277a11ca3f94702" exitCode=0
Sep 16 12:11:26.640034 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:26.639707    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc" event={"ID":"9e5057ea-cc3e-46d4-8223-2c7334b9cc69","Type":"ContainerDied","Data":"3314537bf51d439d0893b4af7405061d520575cb85288bfb0277a11ca3f94702"}
Sep 16 12:11:27.769356 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:27.769330    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc"
Sep 16 12:11:27.855680 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:27.855633    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e5057ea-cc3e-46d4-8223-2c7334b9cc69-util\") pod \"9e5057ea-cc3e-46d4-8223-2c7334b9cc69\" (UID: \"9e5057ea-cc3e-46d4-8223-2c7334b9cc69\") "
Sep 16 12:11:27.855858 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:27.855695    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e5057ea-cc3e-46d4-8223-2c7334b9cc69-bundle\") pod \"9e5057ea-cc3e-46d4-8223-2c7334b9cc69\" (UID: \"9e5057ea-cc3e-46d4-8223-2c7334b9cc69\") "
Sep 16 12:11:27.855858 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:27.855764    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlfsx\" (UniqueName: \"kubernetes.io/projected/9e5057ea-cc3e-46d4-8223-2c7334b9cc69-kube-api-access-mlfsx\") pod \"9e5057ea-cc3e-46d4-8223-2c7334b9cc69\" (UID: \"9e5057ea-cc3e-46d4-8223-2c7334b9cc69\") "
Sep 16 12:11:27.856482 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:27.856454    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5057ea-cc3e-46d4-8223-2c7334b9cc69-bundle" (OuterVolumeSpecName: "bundle") pod "9e5057ea-cc3e-46d4-8223-2c7334b9cc69" (UID: "9e5057ea-cc3e-46d4-8223-2c7334b9cc69"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:11:27.858041 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:27.858014    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5057ea-cc3e-46d4-8223-2c7334b9cc69-kube-api-access-mlfsx" (OuterVolumeSpecName: "kube-api-access-mlfsx") pod "9e5057ea-cc3e-46d4-8223-2c7334b9cc69" (UID: "9e5057ea-cc3e-46d4-8223-2c7334b9cc69"). InnerVolumeSpecName "kube-api-access-mlfsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:11:27.860787 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:27.860763    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5057ea-cc3e-46d4-8223-2c7334b9cc69-util" (OuterVolumeSpecName: "util") pod "9e5057ea-cc3e-46d4-8223-2c7334b9cc69" (UID: "9e5057ea-cc3e-46d4-8223-2c7334b9cc69"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:11:27.956571 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:27.956474    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e5057ea-cc3e-46d4-8223-2c7334b9cc69-util\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:11:27.956571 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:27.956508    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e5057ea-cc3e-46d4-8223-2c7334b9cc69-bundle\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:11:27.956571 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:27.956521    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlfsx\" (UniqueName: \"kubernetes.io/projected/9e5057ea-cc3e-46d4-8223-2c7334b9cc69-kube-api-access-mlfsx\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:11:28.650017 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:28.649989    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc"
Sep 16 12:11:28.650181 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:28.649991    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540md4sc" event={"ID":"9e5057ea-cc3e-46d4-8223-2c7334b9cc69","Type":"ContainerDied","Data":"aea6ef236691af1f47591fbd94dcc2bf34c57bcc9289a52336633c31e8ecdcc3"}
Sep 16 12:11:28.650181 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:28.650105    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aea6ef236691af1f47591fbd94dcc2bf34c57bcc9289a52336633c31e8ecdcc3"
Sep 16 12:11:31.654437 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:31.654398    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hjbvg"]
Sep 16 12:11:31.654834 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:31.654706    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e5057ea-cc3e-46d4-8223-2c7334b9cc69" containerName="extract"
Sep 16 12:11:31.654834 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:31.654717    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e5057ea-cc3e-46d4-8223-2c7334b9cc69" containerName="extract"
Sep 16 12:11:31.654834 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:31.654728    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e5057ea-cc3e-46d4-8223-2c7334b9cc69" containerName="util"
Sep 16 12:11:31.654834 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:31.654734    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e5057ea-cc3e-46d4-8223-2c7334b9cc69" containerName="util"
Sep 16 12:11:31.654834 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:31.654744    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e5057ea-cc3e-46d4-8223-2c7334b9cc69" containerName="pull"
Sep 16 12:11:31.654834 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:31.654750    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e5057ea-cc3e-46d4-8223-2c7334b9cc69" containerName="pull"
Sep 16 12:11:31.654834 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:31.654804    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e5057ea-cc3e-46d4-8223-2c7334b9cc69" containerName="extract"
Sep 16 12:11:31.691199 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:31.691173    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hjbvg"]
Sep 16 12:11:31.691347 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:31.691271    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hjbvg"
Sep 16 12:11:31.693726 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:31.693699    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:11:31.693859 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:31.693768    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-gw74k\""
Sep 16 12:11:31.693859 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:31.693829    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 16 12:11:31.787017 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:31.786983    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6jfv\" (UniqueName: \"kubernetes.io/projected/61c61f27-e158-433d-8fcc-21d159fddb23-kube-api-access-d6jfv\") pod \"cert-manager-operator-controller-manager-55c444bf86-hjbvg\" (UID: \"61c61f27-e158-433d-8fcc-21d159fddb23\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hjbvg"
Sep 16 12:11:31.787179 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:31.787055    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61c61f27-e158-433d-8fcc-21d159fddb23-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hjbvg\" (UID: \"61c61f27-e158-433d-8fcc-21d159fddb23\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hjbvg"
Sep 16 12:11:31.887989 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:31.887953    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6jfv\" (UniqueName: \"kubernetes.io/projected/61c61f27-e158-433d-8fcc-21d159fddb23-kube-api-access-d6jfv\") pod \"cert-manager-operator-controller-manager-55c444bf86-hjbvg\" (UID: \"61c61f27-e158-433d-8fcc-21d159fddb23\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hjbvg"
Sep 16 12:11:31.888150 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:31.888031    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61c61f27-e158-433d-8fcc-21d159fddb23-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hjbvg\" (UID: \"61c61f27-e158-433d-8fcc-21d159fddb23\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hjbvg"
Sep 16 12:11:31.888339 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:31.888323    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61c61f27-e158-433d-8fcc-21d159fddb23-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-hjbvg\" (UID: \"61c61f27-e158-433d-8fcc-21d159fddb23\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hjbvg"
Sep 16 12:11:31.895996 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:31.895973    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6jfv\" (UniqueName: \"kubernetes.io/projected/61c61f27-e158-433d-8fcc-21d159fddb23-kube-api-access-d6jfv\") pod \"cert-manager-operator-controller-manager-55c444bf86-hjbvg\" (UID: \"61c61f27-e158-433d-8fcc-21d159fddb23\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hjbvg"
Sep 16 12:11:31.999962 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:31.999881    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hjbvg"
Sep 16 12:11:32.284761 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:32.284734    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hjbvg"]
Sep 16 12:11:32.287886 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:11:32.287860    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c61f27_e158_433d_8fcc_21d159fddb23.slice/crio-98d087f1fc60b3c8ff05d2b8a0d9071ca24e189f2eeeef2edc0302bb77447fb5 WatchSource:0}: Error finding container 98d087f1fc60b3c8ff05d2b8a0d9071ca24e189f2eeeef2edc0302bb77447fb5: Status 404 returned error can't find the container with id 98d087f1fc60b3c8ff05d2b8a0d9071ca24e189f2eeeef2edc0302bb77447fb5
Sep 16 12:11:32.665993 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:32.665958    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hjbvg" event={"ID":"61c61f27-e158-433d-8fcc-21d159fddb23","Type":"ContainerStarted","Data":"98d087f1fc60b3c8ff05d2b8a0d9071ca24e189f2eeeef2edc0302bb77447fb5"}
Sep 16 12:11:35.680749 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:35.680713    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hjbvg" event={"ID":"61c61f27-e158-433d-8fcc-21d159fddb23","Type":"ContainerStarted","Data":"14df688ef0c11f57fac57fde7795765ee90c66e864c61392199fbae6b1465aba"}
Sep 16 12:11:35.701887 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:35.701831    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-hjbvg" podStartSLOduration=2.264592167 podStartE2EDuration="4.701814906s" totalImagesPullingTime="2.437222739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:11:35.70009415 +0000 UTC m=+489.401165475" watchObservedRunningTime="2026-09-16 12:11:35.701814906 +0000 UTC m=+489.402886231"
Sep 16 12:11:36.709990 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:36.709958    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n"]
Sep 16 12:11:36.712413 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:36.712396    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n"
Sep 16 12:11:36.714791 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:36.714769    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 12:11:36.715736 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:36.715720    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 12:11:36.715810 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:36.715759    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r4zt6\""
Sep 16 12:11:36.720509 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:36.720487    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n"]
Sep 16 12:11:36.829632 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:36.829592    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjbdr\" (UniqueName: \"kubernetes.io/projected/3be0211a-771a-4848-ae39-c2e94d741d22-kube-api-access-zjbdr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n\" (UID: \"3be0211a-771a-4848-ae39-c2e94d741d22\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n"
Sep 16 12:11:36.829837 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:36.829643    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3be0211a-771a-4848-ae39-c2e94d741d22-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n\" (UID: \"3be0211a-771a-4848-ae39-c2e94d741d22\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n"
Sep 16 12:11:36.829837 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:36.829675    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3be0211a-771a-4848-ae39-c2e94d741d22-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n\" (UID: \"3be0211a-771a-4848-ae39-c2e94d741d22\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n"
Sep 16 12:11:36.930969 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:36.930931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjbdr\" (UniqueName: \"kubernetes.io/projected/3be0211a-771a-4848-ae39-c2e94d741d22-kube-api-access-zjbdr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n\" (UID: \"3be0211a-771a-4848-ae39-c2e94d741d22\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n"
Sep 16 12:11:36.930969 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:36.930974    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3be0211a-771a-4848-ae39-c2e94d741d22-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n\" (UID: \"3be0211a-771a-4848-ae39-c2e94d741d22\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n"
Sep 16 12:11:36.931232 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:36.930993    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3be0211a-771a-4848-ae39-c2e94d741d22-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n\" (UID: \"3be0211a-771a-4848-ae39-c2e94d741d22\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n"
Sep 16 12:11:36.931363 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:36.931342    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3be0211a-771a-4848-ae39-c2e94d741d22-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n\" (UID: \"3be0211a-771a-4848-ae39-c2e94d741d22\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n"
Sep 16 12:11:36.931363 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:36.931356    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3be0211a-771a-4848-ae39-c2e94d741d22-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n\" (UID: \"3be0211a-771a-4848-ae39-c2e94d741d22\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n"
Sep 16 12:11:36.938698 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:36.938679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjbdr\" (UniqueName: \"kubernetes.io/projected/3be0211a-771a-4848-ae39-c2e94d741d22-kube-api-access-zjbdr\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n\" (UID: \"3be0211a-771a-4848-ae39-c2e94d741d22\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n"
Sep 16 12:11:37.021462 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:37.021386    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n"
Sep 16 12:11:37.287091 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:37.287020    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n"]
Sep 16 12:11:37.289979 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:11:37.289946    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be0211a_771a_4848_ae39_c2e94d741d22.slice/crio-37cc3efcf2a848045aff0d7b4f56836a679b5ab9341d7b1257adb9772ff560b6 WatchSource:0}: Error finding container 37cc3efcf2a848045aff0d7b4f56836a679b5ab9341d7b1257adb9772ff560b6: Status 404 returned error can't find the container with id 37cc3efcf2a848045aff0d7b4f56836a679b5ab9341d7b1257adb9772ff560b6
Sep 16 12:11:37.690929 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:37.690895    2579 generic.go:356] "Generic (PLEG): container finished" podID="3be0211a-771a-4848-ae39-c2e94d741d22" containerID="c370b077dfc363a4264a3369377dec064786c1f3e4a4d754bf32fef16dc64da6" exitCode=0
Sep 16 12:11:37.691109 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:37.690986    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n" event={"ID":"3be0211a-771a-4848-ae39-c2e94d741d22","Type":"ContainerDied","Data":"c370b077dfc363a4264a3369377dec064786c1f3e4a4d754bf32fef16dc64da6"}
Sep 16 12:11:37.691109 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:37.691045    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n" event={"ID":"3be0211a-771a-4848-ae39-c2e94d741d22","Type":"ContainerStarted","Data":"37cc3efcf2a848045aff0d7b4f56836a679b5ab9341d7b1257adb9772ff560b6"}
Sep 16 12:11:38.277533 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:38.277451    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9rm9j"]
Sep 16 12:11:38.279733 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:38.279717    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9rm9j"
Sep 16 12:11:38.282142 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:38.282121    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 12:11:38.282247 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:38.282132    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 12:11:38.282403 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:38.282390    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-qlt5f\""
Sep 16 12:11:38.289949 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:38.289928    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9rm9j"]
Sep 16 12:11:38.441326 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:38.441289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2wfq\" (UniqueName: \"kubernetes.io/projected/5a041717-5b30-4e2f-8761-70a3c688907f-kube-api-access-z2wfq\") pod \"cert-manager-cainjector-559cdd8c87-9rm9j\" (UID: \"5a041717-5b30-4e2f-8761-70a3c688907f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9rm9j"
Sep 16 12:11:38.441497 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:38.441420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a041717-5b30-4e2f-8761-70a3c688907f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9rm9j\" (UID: \"5a041717-5b30-4e2f-8761-70a3c688907f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9rm9j"
Sep 16 12:11:38.542444 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:38.542358    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a041717-5b30-4e2f-8761-70a3c688907f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9rm9j\" (UID: \"5a041717-5b30-4e2f-8761-70a3c688907f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9rm9j"
Sep 16 12:11:38.542444 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:38.542421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2wfq\" (UniqueName: \"kubernetes.io/projected/5a041717-5b30-4e2f-8761-70a3c688907f-kube-api-access-z2wfq\") pod \"cert-manager-cainjector-559cdd8c87-9rm9j\" (UID: \"5a041717-5b30-4e2f-8761-70a3c688907f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9rm9j"
Sep 16 12:11:38.550831 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:38.550794    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5a041717-5b30-4e2f-8761-70a3c688907f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9rm9j\" (UID: \"5a041717-5b30-4e2f-8761-70a3c688907f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9rm9j"
Sep 16 12:11:38.551047 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:38.551026    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2wfq\" (UniqueName: \"kubernetes.io/projected/5a041717-5b30-4e2f-8761-70a3c688907f-kube-api-access-z2wfq\") pod \"cert-manager-cainjector-559cdd8c87-9rm9j\" (UID: \"5a041717-5b30-4e2f-8761-70a3c688907f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9rm9j"
Sep 16 12:11:38.589807 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:38.589777    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9rm9j"
Sep 16 12:11:38.865074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:38.865009    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9rm9j"]
Sep 16 12:11:38.867071 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:11:38.867045    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a041717_5b30_4e2f_8761_70a3c688907f.slice/crio-c1c7569b1df94e0dd54962035496a4d6b17e3bfc3ba4afdcca9814dae87a7a10 WatchSource:0}: Error finding container c1c7569b1df94e0dd54962035496a4d6b17e3bfc3ba4afdcca9814dae87a7a10: Status 404 returned error can't find the container with id c1c7569b1df94e0dd54962035496a4d6b17e3bfc3ba4afdcca9814dae87a7a10
Sep 16 12:11:39.704135 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:39.704090    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9rm9j" event={"ID":"5a041717-5b30-4e2f-8761-70a3c688907f","Type":"ContainerStarted","Data":"c1c7569b1df94e0dd54962035496a4d6b17e3bfc3ba4afdcca9814dae87a7a10"}
Sep 16 12:11:41.714994 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:41.714965    2579 generic.go:356] "Generic (PLEG): container finished" podID="3be0211a-771a-4848-ae39-c2e94d741d22" containerID="adb00bf2d34c35dad0b4804a0e983a08986533a38cd04de672dc59352761152f" exitCode=0
Sep 16 12:11:41.715375 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:41.715051    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n" event={"ID":"3be0211a-771a-4848-ae39-c2e94d741d22","Type":"ContainerDied","Data":"adb00bf2d34c35dad0b4804a0e983a08986533a38cd04de672dc59352761152f"}
Sep 16 12:11:41.716706 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:41.716677    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9rm9j" event={"ID":"5a041717-5b30-4e2f-8761-70a3c688907f","Type":"ContainerStarted","Data":"70a2e01573a2ce64edb5a3ef3f729cdf673a0547e44c71716d17640dbc5f9423"}
Sep 16 12:11:41.749605 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:41.749558    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9rm9j" podStartSLOduration=1.032036009 podStartE2EDuration="3.749540624s" totalImagesPullingTime="2.717504615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:11:41.748296886 +0000 UTC m=+495.449368213" watchObservedRunningTime="2026-09-16 12:11:41.749540624 +0000 UTC m=+495.450611949"
Sep 16 12:11:42.724492 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:42.724456    2579 generic.go:356] "Generic (PLEG): container finished" podID="3be0211a-771a-4848-ae39-c2e94d741d22" containerID="928705b18b9828875693274545c783156aea4e2a10409fcec55a4d7f3abaab35" exitCode=0
Sep 16 12:11:42.724992 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:42.724546    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n" event={"ID":"3be0211a-771a-4848-ae39-c2e94d741d22","Type":"ContainerDied","Data":"928705b18b9828875693274545c783156aea4e2a10409fcec55a4d7f3abaab35"}
Sep 16 12:11:43.860349 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:43.860320    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n"
Sep 16 12:11:43.889006 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:43.888789    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjbdr\" (UniqueName: \"kubernetes.io/projected/3be0211a-771a-4848-ae39-c2e94d741d22-kube-api-access-zjbdr\") pod \"3be0211a-771a-4848-ae39-c2e94d741d22\" (UID: \"3be0211a-771a-4848-ae39-c2e94d741d22\") "
Sep 16 12:11:43.889182 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:43.889051    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3be0211a-771a-4848-ae39-c2e94d741d22-util\") pod \"3be0211a-771a-4848-ae39-c2e94d741d22\" (UID: \"3be0211a-771a-4848-ae39-c2e94d741d22\") "
Sep 16 12:11:43.889182 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:43.889089    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3be0211a-771a-4848-ae39-c2e94d741d22-bundle\") pod \"3be0211a-771a-4848-ae39-c2e94d741d22\" (UID: \"3be0211a-771a-4848-ae39-c2e94d741d22\") "
Sep 16 12:11:43.889626 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:43.889597    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be0211a-771a-4848-ae39-c2e94d741d22-bundle" (OuterVolumeSpecName: "bundle") pod "3be0211a-771a-4848-ae39-c2e94d741d22" (UID: "3be0211a-771a-4848-ae39-c2e94d741d22"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:11:43.891166 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:43.891142    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be0211a-771a-4848-ae39-c2e94d741d22-kube-api-access-zjbdr" (OuterVolumeSpecName: "kube-api-access-zjbdr") pod "3be0211a-771a-4848-ae39-c2e94d741d22" (UID: "3be0211a-771a-4848-ae39-c2e94d741d22"). InnerVolumeSpecName "kube-api-access-zjbdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:11:43.895744 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:43.895695    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be0211a-771a-4848-ae39-c2e94d741d22-util" (OuterVolumeSpecName: "util") pod "3be0211a-771a-4848-ae39-c2e94d741d22" (UID: "3be0211a-771a-4848-ae39-c2e94d741d22"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:11:43.990180 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:43.990093    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjbdr\" (UniqueName: \"kubernetes.io/projected/3be0211a-771a-4848-ae39-c2e94d741d22-kube-api-access-zjbdr\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:11:43.990180 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:43.990123    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3be0211a-771a-4848-ae39-c2e94d741d22-util\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:11:43.990180 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:43.990134    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3be0211a-771a-4848-ae39-c2e94d741d22-bundle\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:11:44.734831 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:44.734803    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n"
Sep 16 12:11:44.734999 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:44.734833    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgt45n" event={"ID":"3be0211a-771a-4848-ae39-c2e94d741d22","Type":"ContainerDied","Data":"37cc3efcf2a848045aff0d7b4f56836a679b5ab9341d7b1257adb9772ff560b6"}
Sep 16 12:11:44.734999 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:44.734901    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37cc3efcf2a848045aff0d7b4f56836a679b5ab9341d7b1257adb9772ff560b6"
Sep 16 12:11:46.968376 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:46.968332    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vxc86"]
Sep 16 12:11:46.968867 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:46.968775    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3be0211a-771a-4848-ae39-c2e94d741d22" containerName="extract"
Sep 16 12:11:46.968867 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:46.968793    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3be0211a-771a-4848-ae39-c2e94d741d22" containerName="extract"
Sep 16 12:11:46.968867 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:46.968833    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3be0211a-771a-4848-ae39-c2e94d741d22" containerName="util"
Sep 16 12:11:46.968867 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:46.968841    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3be0211a-771a-4848-ae39-c2e94d741d22" containerName="util"
Sep 16 12:11:46.968867 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:46.968853    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3be0211a-771a-4848-ae39-c2e94d741d22" containerName="pull"
Sep 16 12:11:46.968867 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:46.968861    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3be0211a-771a-4848-ae39-c2e94d741d22" containerName="pull"
Sep 16 12:11:46.969162 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:46.968961    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3be0211a-771a-4848-ae39-c2e94d741d22" containerName="extract"
Sep 16 12:11:46.970875 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:46.970856    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-vxc86"
Sep 16 12:11:46.973216 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:46.973195    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-p2tfr\""
Sep 16 12:11:46.980797 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:46.980774    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vxc86"]
Sep 16 12:11:47.013641 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:47.013604    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b519f3b0-5f41-4860-938c-f1de53038c49-bound-sa-token\") pod \"cert-manager-55b4d954d5-vxc86\" (UID: \"b519f3b0-5f41-4860-938c-f1de53038c49\") " pod="cert-manager/cert-manager-55b4d954d5-vxc86"
Sep 16 12:11:47.013641 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:47.013649    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5v75\" (UniqueName: \"kubernetes.io/projected/b519f3b0-5f41-4860-938c-f1de53038c49-kube-api-access-m5v75\") pod \"cert-manager-55b4d954d5-vxc86\" (UID: \"b519f3b0-5f41-4860-938c-f1de53038c49\") " pod="cert-manager/cert-manager-55b4d954d5-vxc86"
Sep 16 12:11:47.114346 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:47.114308    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b519f3b0-5f41-4860-938c-f1de53038c49-bound-sa-token\") pod \"cert-manager-55b4d954d5-vxc86\" (UID: \"b519f3b0-5f41-4860-938c-f1de53038c49\") " pod="cert-manager/cert-manager-55b4d954d5-vxc86"
Sep 16 12:11:47.114519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:47.114373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5v75\" (UniqueName: \"kubernetes.io/projected/b519f3b0-5f41-4860-938c-f1de53038c49-kube-api-access-m5v75\") pod \"cert-manager-55b4d954d5-vxc86\" (UID: \"b519f3b0-5f41-4860-938c-f1de53038c49\") " pod="cert-manager/cert-manager-55b4d954d5-vxc86"
Sep 16 12:11:47.122458 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:47.122421    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b519f3b0-5f41-4860-938c-f1de53038c49-bound-sa-token\") pod \"cert-manager-55b4d954d5-vxc86\" (UID: \"b519f3b0-5f41-4860-938c-f1de53038c49\") " pod="cert-manager/cert-manager-55b4d954d5-vxc86"
Sep 16 12:11:47.122458 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:47.122440    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5v75\" (UniqueName: \"kubernetes.io/projected/b519f3b0-5f41-4860-938c-f1de53038c49-kube-api-access-m5v75\") pod \"cert-manager-55b4d954d5-vxc86\" (UID: \"b519f3b0-5f41-4860-938c-f1de53038c49\") " pod="cert-manager/cert-manager-55b4d954d5-vxc86"
Sep 16 12:11:47.280498 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:47.280408    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-vxc86"
Sep 16 12:11:47.547410 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:47.547296    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vxc86"]
Sep 16 12:11:47.549643 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:11:47.549616    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb519f3b0_5f41_4860_938c_f1de53038c49.slice/crio-dc54646126258b1c696eefbcf1786fa71ac2ad78b965bfb2a751bdf191f8c92c WatchSource:0}: Error finding container dc54646126258b1c696eefbcf1786fa71ac2ad78b965bfb2a751bdf191f8c92c: Status 404 returned error can't find the container with id dc54646126258b1c696eefbcf1786fa71ac2ad78b965bfb2a751bdf191f8c92c
Sep 16 12:11:47.748522 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:47.748483    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vxc86" event={"ID":"b519f3b0-5f41-4860-938c-f1de53038c49","Type":"ContainerStarted","Data":"f012407fec2cee4cfc28cb28818b973b06dd8a2fca4525df3ac66870ef60a794"}
Sep 16 12:11:47.748522 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:47.748528    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vxc86" event={"ID":"b519f3b0-5f41-4860-938c-f1de53038c49","Type":"ContainerStarted","Data":"dc54646126258b1c696eefbcf1786fa71ac2ad78b965bfb2a751bdf191f8c92c"}
Sep 16 12:11:47.764279 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:47.764225    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-vxc86" podStartSLOduration=1.764207378 podStartE2EDuration="1.764207378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:11:47.762445614 +0000 UTC m=+501.463516952" watchObservedRunningTime="2026-09-16 12:11:47.764207378 +0000 UTC m=+501.465278705"
Sep 16 12:11:54.892733 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:54.892697    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlxxw"]
Sep 16 12:11:54.894978 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:54.894962    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlxxw"
Sep 16 12:11:54.897257 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:54.897231    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 12:11:54.897257 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:54.897242    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 12:11:54.898242 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:54.898224    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-wdb66\""
Sep 16 12:11:54.903498 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:54.903474    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlxxw"]
Sep 16 12:11:54.973887 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:54.973850    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c376e5cf-86e5-4bed-b75b-70c39ad18cac-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wlxxw\" (UID: \"c376e5cf-86e5-4bed-b75b-70c39ad18cac\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlxxw"
Sep 16 12:11:54.974054 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:54.973924    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v57w9\" (UniqueName: \"kubernetes.io/projected/c376e5cf-86e5-4bed-b75b-70c39ad18cac-kube-api-access-v57w9\") pod \"openshift-lws-operator-fd8ccff4c-wlxxw\" (UID: \"c376e5cf-86e5-4bed-b75b-70c39ad18cac\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlxxw"
Sep 16 12:11:55.074266 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:55.074237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v57w9\" (UniqueName: \"kubernetes.io/projected/c376e5cf-86e5-4bed-b75b-70c39ad18cac-kube-api-access-v57w9\") pod \"openshift-lws-operator-fd8ccff4c-wlxxw\" (UID: \"c376e5cf-86e5-4bed-b75b-70c39ad18cac\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlxxw"
Sep 16 12:11:55.074399 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:55.074286    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c376e5cf-86e5-4bed-b75b-70c39ad18cac-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wlxxw\" (UID: \"c376e5cf-86e5-4bed-b75b-70c39ad18cac\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlxxw"
Sep 16 12:11:55.074628 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:55.074605    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c376e5cf-86e5-4bed-b75b-70c39ad18cac-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wlxxw\" (UID: \"c376e5cf-86e5-4bed-b75b-70c39ad18cac\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlxxw"
Sep 16 12:11:55.081612 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:55.081592    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v57w9\" (UniqueName: \"kubernetes.io/projected/c376e5cf-86e5-4bed-b75b-70c39ad18cac-kube-api-access-v57w9\") pod \"openshift-lws-operator-fd8ccff4c-wlxxw\" (UID: \"c376e5cf-86e5-4bed-b75b-70c39ad18cac\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlxxw"
Sep 16 12:11:55.204621 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:55.204523    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlxxw"
Sep 16 12:11:55.469880 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:55.469797    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlxxw"]
Sep 16 12:11:55.472772 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:11:55.472746    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc376e5cf_86e5_4bed_b75b_70c39ad18cac.slice/crio-a67580ec6a168abf68049090a45220f51c9dbc7c64d254b4fc223e7f045271e9 WatchSource:0}: Error finding container a67580ec6a168abf68049090a45220f51c9dbc7c64d254b4fc223e7f045271e9: Status 404 returned error can't find the container with id a67580ec6a168abf68049090a45220f51c9dbc7c64d254b4fc223e7f045271e9
Sep 16 12:11:55.789737 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:55.789633    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlxxw" event={"ID":"c376e5cf-86e5-4bed-b75b-70c39ad18cac","Type":"ContainerStarted","Data":"a67580ec6a168abf68049090a45220f51c9dbc7c64d254b4fc223e7f045271e9"}
Sep 16 12:11:58.805403 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:58.805366    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlxxw" event={"ID":"c376e5cf-86e5-4bed-b75b-70c39ad18cac","Type":"ContainerStarted","Data":"838b423cf93b51965d9dbc263dbcdf4ce0735e2e9644e20852d935f4740a91b2"}
Sep 16 12:11:58.822952 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:11:58.822898    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wlxxw" podStartSLOduration=2.37909363 podStartE2EDuration="4.822881784s" totalImagesPullingTime="2.443788154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:11:58.821303559 +0000 UTC m=+512.522374886" watchObservedRunningTime="2026-09-16 12:11:58.822881784 +0000 UTC m=+512.523953109"
Sep 16 12:12:01.486591 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:01.486549    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l"]
Sep 16 12:12:01.489484 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:01.489456    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l"
Sep 16 12:12:01.492235 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:01.492209    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 12:12:01.492379 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:01.492275    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 12:12:01.493136 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:01.493119    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r4zt6\""
Sep 16 12:12:01.502425 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:01.502397    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l"]
Sep 16 12:12:01.525829 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:01.525793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg5jg\" (UniqueName: \"kubernetes.io/projected/a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4-kube-api-access-wg5jg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l\" (UID: \"a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l"
Sep 16 12:12:01.525989 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:01.525898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l\" (UID: \"a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l"
Sep 16 12:12:01.525989 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:01.525932    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l\" (UID: \"a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l"
Sep 16 12:12:01.627235 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:01.627198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l\" (UID: \"a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l"
Sep 16 12:12:01.627520 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:01.627244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l\" (UID: \"a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l"
Sep 16 12:12:01.627520 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:01.627267    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg5jg\" (UniqueName: \"kubernetes.io/projected/a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4-kube-api-access-wg5jg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l\" (UID: \"a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l"
Sep 16 12:12:01.627757 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:01.627731    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l\" (UID: \"a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l"
Sep 16 12:12:01.627829 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:01.627742    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l\" (UID: \"a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l"
Sep 16 12:12:01.635384 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:01.635355    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg5jg\" (UniqueName: \"kubernetes.io/projected/a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4-kube-api-access-wg5jg\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l\" (UID: \"a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l"
Sep 16 12:12:01.799179 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:01.799097    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l"
Sep 16 12:12:02.071089 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:02.071058    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l"]
Sep 16 12:12:02.073777 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:12:02.073750    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda90fd2a4_7a2e_4018_b1be_17fa44c3fbe4.slice/crio-2844160c55f6b101a7ba02d415bca584fe6121de9fec2c032f38ed2da3231e70 WatchSource:0}: Error finding container 2844160c55f6b101a7ba02d415bca584fe6121de9fec2c032f38ed2da3231e70: Status 404 returned error can't find the container with id 2844160c55f6b101a7ba02d415bca584fe6121de9fec2c032f38ed2da3231e70
Sep 16 12:12:02.828345 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:02.828314    2579 generic.go:356] "Generic (PLEG): container finished" podID="a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4" containerID="72defe325bbfc6d6db73f1b7cfba0aaeb0d82ab87f40c15ffa91e852fa014643" exitCode=0
Sep 16 12:12:02.828779 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:02.828385    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l" event={"ID":"a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4","Type":"ContainerDied","Data":"72defe325bbfc6d6db73f1b7cfba0aaeb0d82ab87f40c15ffa91e852fa014643"}
Sep 16 12:12:02.828779 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:02.828450    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l" event={"ID":"a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4","Type":"ContainerStarted","Data":"2844160c55f6b101a7ba02d415bca584fe6121de9fec2c032f38ed2da3231e70"}
Sep 16 12:12:03.835384 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:03.835350    2579 generic.go:356] "Generic (PLEG): container finished" podID="a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4" containerID="06a5622317c9db236226ef97d0dfc9193eb928c927b98639e763597275566d27" exitCode=0
Sep 16 12:12:03.835767 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:03.835438    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l" event={"ID":"a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4","Type":"ContainerDied","Data":"06a5622317c9db236226ef97d0dfc9193eb928c927b98639e763597275566d27"}
Sep 16 12:12:04.842372 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:04.842334    2579 generic.go:356] "Generic (PLEG): container finished" podID="a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4" containerID="db5ec15c73c866997587573b3ade2a2811081463b91d9f175797c4785472bb26" exitCode=0
Sep 16 12:12:04.842790 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:04.842424    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l" event={"ID":"a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4","Type":"ContainerDied","Data":"db5ec15c73c866997587573b3ade2a2811081463b91d9f175797c4785472bb26"}
Sep 16 12:12:05.972869 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:05.972845    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l"
Sep 16 12:12:06.069466 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:06.069429    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wg5jg\" (UniqueName: \"kubernetes.io/projected/a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4-kube-api-access-wg5jg\") pod \"a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4\" (UID: \"a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4\") "
Sep 16 12:12:06.069679 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:06.069490    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4-util\") pod \"a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4\" (UID: \"a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4\") "
Sep 16 12:12:06.069679 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:06.069620    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4-bundle\") pod \"a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4\" (UID: \"a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4\") "
Sep 16 12:12:06.070439 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:06.070415    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4-bundle" (OuterVolumeSpecName: "bundle") pod "a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4" (UID: "a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:12:06.071611 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:06.071590    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4-kube-api-access-wg5jg" (OuterVolumeSpecName: "kube-api-access-wg5jg") pod "a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4" (UID: "a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4"). InnerVolumeSpecName "kube-api-access-wg5jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:12:06.075369 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:06.075350    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4-util" (OuterVolumeSpecName: "util") pod "a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4" (UID: "a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:12:06.170515 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:06.170422    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg5jg\" (UniqueName: \"kubernetes.io/projected/a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4-kube-api-access-wg5jg\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:12:06.170515 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:06.170456    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4-util\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:12:06.170515 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:06.170465    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4-bundle\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:12:06.854016 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:06.853988    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l"
Sep 16 12:12:06.854174 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:06.853986    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afxq26l" event={"ID":"a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4","Type":"ContainerDied","Data":"2844160c55f6b101a7ba02d415bca584fe6121de9fec2c032f38ed2da3231e70"}
Sep 16 12:12:06.854174 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:06.854106    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2844160c55f6b101a7ba02d415bca584fe6121de9fec2c032f38ed2da3231e70"
Sep 16 12:12:09.949460 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:09.949425    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-74d55495b7-pcng2"]
Sep 16 12:12:09.949879 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:09.949754    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4" containerName="extract"
Sep 16 12:12:09.949879 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:09.949766    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4" containerName="extract"
Sep 16 12:12:09.949879 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:09.949775    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4" containerName="pull"
Sep 16 12:12:09.949879 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:09.949780    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4" containerName="pull"
Sep 16 12:12:09.949879 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:09.949792    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4" containerName="util"
Sep 16 12:12:09.949879 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:09.949798    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4" containerName="util"
Sep 16 12:12:09.949879 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:09.949851    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a90fd2a4-7a2e-4018-b1be-17fa44c3fbe4" containerName="extract"
Sep 16 12:12:09.952492 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:09.952476    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74d55495b7-pcng2"
Sep 16 12:12:09.956169 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:09.956147    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rhnbv\""
Sep 16 12:12:09.956298 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:09.956214    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 12:12:09.956298 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:09.956236    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 12:12:09.956298 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:09.956238    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 12:12:09.965481 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:09.965458    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74d55495b7-pcng2"]
Sep 16 12:12:09.999182 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:09.999144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13e5e417-b667-4c12-ab6c-d0c3aafb6eb0-cert\") pod \"lws-controller-manager-74d55495b7-pcng2\" (UID: \"13e5e417-b667-4c12-ab6c-d0c3aafb6eb0\") " pod="openshift-lws-operator/lws-controller-manager-74d55495b7-pcng2"
Sep 16 12:12:09.999324 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:09.999196    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/13e5e417-b667-4c12-ab6c-d0c3aafb6eb0-manager-config\") pod \"lws-controller-manager-74d55495b7-pcng2\" (UID: \"13e5e417-b667-4c12-ab6c-d0c3aafb6eb0\") " pod="openshift-lws-operator/lws-controller-manager-74d55495b7-pcng2"
Sep 16 12:12:09.999324 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:09.999219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/13e5e417-b667-4c12-ab6c-d0c3aafb6eb0-metrics-cert\") pod \"lws-controller-manager-74d55495b7-pcng2\" (UID: \"13e5e417-b667-4c12-ab6c-d0c3aafb6eb0\") " pod="openshift-lws-operator/lws-controller-manager-74d55495b7-pcng2"
Sep 16 12:12:09.999324 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:09.999251    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-846qv\" (UniqueName: \"kubernetes.io/projected/13e5e417-b667-4c12-ab6c-d0c3aafb6eb0-kube-api-access-846qv\") pod \"lws-controller-manager-74d55495b7-pcng2\" (UID: \"13e5e417-b667-4c12-ab6c-d0c3aafb6eb0\") " pod="openshift-lws-operator/lws-controller-manager-74d55495b7-pcng2"
Sep 16 12:12:10.099708 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:10.099647    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13e5e417-b667-4c12-ab6c-d0c3aafb6eb0-cert\") pod \"lws-controller-manager-74d55495b7-pcng2\" (UID: \"13e5e417-b667-4c12-ab6c-d0c3aafb6eb0\") " pod="openshift-lws-operator/lws-controller-manager-74d55495b7-pcng2"
Sep 16 12:12:10.099879 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:10.099716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/13e5e417-b667-4c12-ab6c-d0c3aafb6eb0-manager-config\") pod \"lws-controller-manager-74d55495b7-pcng2\" (UID: \"13e5e417-b667-4c12-ab6c-d0c3aafb6eb0\") " pod="openshift-lws-operator/lws-controller-manager-74d55495b7-pcng2"
Sep 16 12:12:10.099879 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:10.099734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/13e5e417-b667-4c12-ab6c-d0c3aafb6eb0-metrics-cert\") pod \"lws-controller-manager-74d55495b7-pcng2\" (UID: \"13e5e417-b667-4c12-ab6c-d0c3aafb6eb0\") " pod="openshift-lws-operator/lws-controller-manager-74d55495b7-pcng2"
Sep 16 12:12:10.099879 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:10.099758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-846qv\" (UniqueName: \"kubernetes.io/projected/13e5e417-b667-4c12-ab6c-d0c3aafb6eb0-kube-api-access-846qv\") pod \"lws-controller-manager-74d55495b7-pcng2\" (UID: \"13e5e417-b667-4c12-ab6c-d0c3aafb6eb0\") " pod="openshift-lws-operator/lws-controller-manager-74d55495b7-pcng2"
Sep 16 12:12:10.100398 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:10.100374    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/13e5e417-b667-4c12-ab6c-d0c3aafb6eb0-manager-config\") pod \"lws-controller-manager-74d55495b7-pcng2\" (UID: \"13e5e417-b667-4c12-ab6c-d0c3aafb6eb0\") " pod="openshift-lws-operator/lws-controller-manager-74d55495b7-pcng2"
Sep 16 12:12:10.102288 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:10.102262    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13e5e417-b667-4c12-ab6c-d0c3aafb6eb0-cert\") pod \"lws-controller-manager-74d55495b7-pcng2\" (UID: \"13e5e417-b667-4c12-ab6c-d0c3aafb6eb0\") " pod="openshift-lws-operator/lws-controller-manager-74d55495b7-pcng2"
Sep 16 12:12:10.102380 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:10.102306    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/13e5e417-b667-4c12-ab6c-d0c3aafb6eb0-metrics-cert\") pod \"lws-controller-manager-74d55495b7-pcng2\" (UID: \"13e5e417-b667-4c12-ab6c-d0c3aafb6eb0\") " pod="openshift-lws-operator/lws-controller-manager-74d55495b7-pcng2"
Sep 16 12:12:10.114166 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:10.114144    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-846qv\" (UniqueName: \"kubernetes.io/projected/13e5e417-b667-4c12-ab6c-d0c3aafb6eb0-kube-api-access-846qv\") pod \"lws-controller-manager-74d55495b7-pcng2\" (UID: \"13e5e417-b667-4c12-ab6c-d0c3aafb6eb0\") " pod="openshift-lws-operator/lws-controller-manager-74d55495b7-pcng2"
Sep 16 12:12:10.262225 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:10.262137    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74d55495b7-pcng2"
Sep 16 12:12:10.533898 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:10.533872    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74d55495b7-pcng2"]
Sep 16 12:12:10.536308 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:12:10.536285    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e5e417_b667_4c12_ab6c_d0c3aafb6eb0.slice/crio-a8809bc206ceecbbe623c249145759d8fd6cd4c036d54b4a1b53984b984948da WatchSource:0}: Error finding container a8809bc206ceecbbe623c249145759d8fd6cd4c036d54b4a1b53984b984948da: Status 404 returned error can't find the container with id a8809bc206ceecbbe623c249145759d8fd6cd4c036d54b4a1b53984b984948da
Sep 16 12:12:10.874696 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:10.874640    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74d55495b7-pcng2" event={"ID":"13e5e417-b667-4c12-ab6c-d0c3aafb6eb0","Type":"ContainerStarted","Data":"a8809bc206ceecbbe623c249145759d8fd6cd4c036d54b4a1b53984b984948da"}
Sep 16 12:12:12.093486 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:12.093456    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c"]
Sep 16 12:12:12.096200 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:12.096178    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c"
Sep 16 12:12:12.100370 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:12.100318    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 12:12:12.100370 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:12.100325    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r4zt6\""
Sep 16 12:12:12.101263 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:12.101244    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 12:12:12.109151 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:12.109126    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c"]
Sep 16 12:12:12.217749 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:12.217714    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d84d6f7-9cf6-4696-8b77-ba70bf50c12b-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c\" (UID: \"2d84d6f7-9cf6-4696-8b77-ba70bf50c12b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c"
Sep 16 12:12:12.217921 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:12.217763    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-258gg\" (UniqueName: \"kubernetes.io/projected/2d84d6f7-9cf6-4696-8b77-ba70bf50c12b-kube-api-access-258gg\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c\" (UID: \"2d84d6f7-9cf6-4696-8b77-ba70bf50c12b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c"
Sep 16 12:12:12.217921 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:12.217848    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d84d6f7-9cf6-4696-8b77-ba70bf50c12b-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c\" (UID: \"2d84d6f7-9cf6-4696-8b77-ba70bf50c12b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c"
Sep 16 12:12:12.318761 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:12.318723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d84d6f7-9cf6-4696-8b77-ba70bf50c12b-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c\" (UID: \"2d84d6f7-9cf6-4696-8b77-ba70bf50c12b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c"
Sep 16 12:12:12.318923 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:12.318789    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d84d6f7-9cf6-4696-8b77-ba70bf50c12b-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c\" (UID: \"2d84d6f7-9cf6-4696-8b77-ba70bf50c12b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c"
Sep 16 12:12:12.318923 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:12.318819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-258gg\" (UniqueName: \"kubernetes.io/projected/2d84d6f7-9cf6-4696-8b77-ba70bf50c12b-kube-api-access-258gg\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c\" (UID: \"2d84d6f7-9cf6-4696-8b77-ba70bf50c12b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c"
Sep 16 12:12:12.319107 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:12.319087    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d84d6f7-9cf6-4696-8b77-ba70bf50c12b-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c\" (UID: \"2d84d6f7-9cf6-4696-8b77-ba70bf50c12b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c"
Sep 16 12:12:12.319180 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:12.319159    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d84d6f7-9cf6-4696-8b77-ba70bf50c12b-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c\" (UID: \"2d84d6f7-9cf6-4696-8b77-ba70bf50c12b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c"
Sep 16 12:12:12.339903 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:12.339870    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-258gg\" (UniqueName: \"kubernetes.io/projected/2d84d6f7-9cf6-4696-8b77-ba70bf50c12b-kube-api-access-258gg\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c\" (UID: \"2d84d6f7-9cf6-4696-8b77-ba70bf50c12b\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c"
Sep 16 12:12:12.406428 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:12.406336    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c"
Sep 16 12:12:12.679530 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:12.679383    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c"]
Sep 16 12:12:12.682120 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:12:12.682089    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d84d6f7_9cf6_4696_8b77_ba70bf50c12b.slice/crio-bcefd37a5b0442b470671ae6fc2892d2a74a90e335f87a7d6100ff55afa2e67a WatchSource:0}: Error finding container bcefd37a5b0442b470671ae6fc2892d2a74a90e335f87a7d6100ff55afa2e67a: Status 404 returned error can't find the container with id bcefd37a5b0442b470671ae6fc2892d2a74a90e335f87a7d6100ff55afa2e67a
Sep 16 12:12:12.887960 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:12.887926    2579 generic.go:356] "Generic (PLEG): container finished" podID="2d84d6f7-9cf6-4696-8b77-ba70bf50c12b" containerID="e1f0487b71d326e59624c7d29e893dec7ad7b6ef8d8e2d5644a678aa58930846" exitCode=0
Sep 16 12:12:12.888167 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:12.887988    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c" event={"ID":"2d84d6f7-9cf6-4696-8b77-ba70bf50c12b","Type":"ContainerDied","Data":"e1f0487b71d326e59624c7d29e893dec7ad7b6ef8d8e2d5644a678aa58930846"}
Sep 16 12:12:12.888167 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:12.888061    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c" event={"ID":"2d84d6f7-9cf6-4696-8b77-ba70bf50c12b","Type":"ContainerStarted","Data":"bcefd37a5b0442b470671ae6fc2892d2a74a90e335f87a7d6100ff55afa2e67a"}
Sep 16 12:12:12.889765 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:12.889742    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74d55495b7-pcng2" event={"ID":"13e5e417-b667-4c12-ab6c-d0c3aafb6eb0","Type":"ContainerStarted","Data":"718e6e9b7f04958503e50b1ce173b44af3a9c7856cf45c477b0ac85a01de26e0"}
Sep 16 12:12:12.889903 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:12.889889    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-74d55495b7-pcng2"
Sep 16 12:12:12.976349 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:12.976235    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-74d55495b7-pcng2" podStartSLOduration=2.6624931309999997 podStartE2EDuration="3.976218396s" totalImagesPullingTime="1.313725265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:12:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:12:12.975102107 +0000 UTC m=+526.676173436" watchObservedRunningTime="2026-09-16 12:12:12.976218396 +0000 UTC m=+526.677289721"
Sep 16 12:12:13.895727 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:13.895689    2579 generic.go:356] "Generic (PLEG): container finished" podID="2d84d6f7-9cf6-4696-8b77-ba70bf50c12b" containerID="5798c950157e39e2bbc3165644014981841157c429bed4172902f64504729239" exitCode=0
Sep 16 12:12:13.896173 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:13.895772    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c" event={"ID":"2d84d6f7-9cf6-4696-8b77-ba70bf50c12b","Type":"ContainerDied","Data":"5798c950157e39e2bbc3165644014981841157c429bed4172902f64504729239"}
Sep 16 12:12:14.413705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.413670    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x"]
Sep 16 12:12:14.445250 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.445216    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x"
Sep 16 12:12:14.448369 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.448344    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 12:12:14.448612 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.448597    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-nps9s\""
Sep 16 12:12:14.448710 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.448613    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 12:12:14.448903 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.448885    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 12:12:14.448971 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.448916    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 12:12:14.457543 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.457519    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x"]
Sep 16 12:12:14.535998 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.535964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4-tmp\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-2446x\" (UID: \"c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x"
Sep 16 12:12:14.536162 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.536010    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-2446x\" (UID: \"c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x"
Sep 16 12:12:14.536162 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.536027    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-2446x\" (UID: \"c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x"
Sep 16 12:12:14.536162 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.536111    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9x9j\" (UniqueName: \"kubernetes.io/projected/c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4-kube-api-access-l9x9j\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-2446x\" (UID: \"c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x"
Sep 16 12:12:14.536162 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.536137    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4-manifests\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-2446x\" (UID: \"c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x"
Sep 16 12:12:14.637230 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.637195    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9x9j\" (UniqueName: \"kubernetes.io/projected/c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4-kube-api-access-l9x9j\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-2446x\" (UID: \"c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x"
Sep 16 12:12:14.637409 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.637237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4-manifests\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-2446x\" (UID: \"c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x"
Sep 16 12:12:14.637409 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.637294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4-tmp\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-2446x\" (UID: \"c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x"
Sep 16 12:12:14.637409 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.637336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-2446x\" (UID: \"c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x"
Sep 16 12:12:14.637409 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.637357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-2446x\" (UID: \"c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x"
Sep 16 12:12:14.637800 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.637776    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4-manifests\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-2446x\" (UID: \"c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x"
Sep 16 12:12:14.639605 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.639578    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4-tmp\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-2446x\" (UID: \"c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x"
Sep 16 12:12:14.639930 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.639911    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-2446x\" (UID: \"c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x"
Sep 16 12:12:14.639976 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.639936    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-2446x\" (UID: \"c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x"
Sep 16 12:12:14.645852 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.645826    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9x9j\" (UniqueName: \"kubernetes.io/projected/c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4-kube-api-access-l9x9j\") pod \"opendatahub-operator-controller-manager-6b5b57bf8b-2446x\" (UID: \"c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4\") " pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x"
Sep 16 12:12:14.756192 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.756099    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x"
Sep 16 12:12:14.907965 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.907927    2579 generic.go:356] "Generic (PLEG): container finished" podID="2d84d6f7-9cf6-4696-8b77-ba70bf50c12b" containerID="a161a352796a5a618f311e7a9c43c845af7e58f3265179f30b36e4d2e8d2c106" exitCode=0
Sep 16 12:12:14.908326 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:14.907987    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c" event={"ID":"2d84d6f7-9cf6-4696-8b77-ba70bf50c12b","Type":"ContainerDied","Data":"a161a352796a5a618f311e7a9c43c845af7e58f3265179f30b36e4d2e8d2c106"}
Sep 16 12:12:15.050014 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:15.049973    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x"]
Sep 16 12:12:15.052354 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:12:15.052325    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e3d369_b66e_41f3_8f39_4d90cfd9fdc4.slice/crio-37d608c65331b3e23c451d2a19b90090aad9d1d14ee36da726f6f0b9c4dbb5e2 WatchSource:0}: Error finding container 37d608c65331b3e23c451d2a19b90090aad9d1d14ee36da726f6f0b9c4dbb5e2: Status 404 returned error can't find the container with id 37d608c65331b3e23c451d2a19b90090aad9d1d14ee36da726f6f0b9c4dbb5e2
Sep 16 12:12:15.917284 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:15.917245    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x" event={"ID":"c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4","Type":"ContainerStarted","Data":"37d608c65331b3e23c451d2a19b90090aad9d1d14ee36da726f6f0b9c4dbb5e2"}
Sep 16 12:12:16.078539 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:16.078509    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c"
Sep 16 12:12:16.152633 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:16.152592    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d84d6f7-9cf6-4696-8b77-ba70bf50c12b-util\") pod \"2d84d6f7-9cf6-4696-8b77-ba70bf50c12b\" (UID: \"2d84d6f7-9cf6-4696-8b77-ba70bf50c12b\") "
Sep 16 12:12:16.152831 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:16.152671    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-258gg\" (UniqueName: \"kubernetes.io/projected/2d84d6f7-9cf6-4696-8b77-ba70bf50c12b-kube-api-access-258gg\") pod \"2d84d6f7-9cf6-4696-8b77-ba70bf50c12b\" (UID: \"2d84d6f7-9cf6-4696-8b77-ba70bf50c12b\") "
Sep 16 12:12:16.152831 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:16.152781    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d84d6f7-9cf6-4696-8b77-ba70bf50c12b-bundle\") pod \"2d84d6f7-9cf6-4696-8b77-ba70bf50c12b\" (UID: \"2d84d6f7-9cf6-4696-8b77-ba70bf50c12b\") "
Sep 16 12:12:16.153775 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:16.153729    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d84d6f7-9cf6-4696-8b77-ba70bf50c12b-bundle" (OuterVolumeSpecName: "bundle") pod "2d84d6f7-9cf6-4696-8b77-ba70bf50c12b" (UID: "2d84d6f7-9cf6-4696-8b77-ba70bf50c12b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:12:16.155530 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:16.155502    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d84d6f7-9cf6-4696-8b77-ba70bf50c12b-kube-api-access-258gg" (OuterVolumeSpecName: "kube-api-access-258gg") pod "2d84d6f7-9cf6-4696-8b77-ba70bf50c12b" (UID: "2d84d6f7-9cf6-4696-8b77-ba70bf50c12b"). InnerVolumeSpecName "kube-api-access-258gg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:12:16.160700 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:16.160673    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d84d6f7-9cf6-4696-8b77-ba70bf50c12b-util" (OuterVolumeSpecName: "util") pod "2d84d6f7-9cf6-4696-8b77-ba70bf50c12b" (UID: "2d84d6f7-9cf6-4696-8b77-ba70bf50c12b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:12:16.254189 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:16.254102    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d84d6f7-9cf6-4696-8b77-ba70bf50c12b-bundle\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:12:16.254189 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:16.254139    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d84d6f7-9cf6-4696-8b77-ba70bf50c12b-util\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:12:16.254189 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:16.254153    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-258gg\" (UniqueName: \"kubernetes.io/projected/2d84d6f7-9cf6-4696-8b77-ba70bf50c12b-kube-api-access-258gg\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:12:16.924083 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:16.924050    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c" event={"ID":"2d84d6f7-9cf6-4696-8b77-ba70bf50c12b","Type":"ContainerDied","Data":"bcefd37a5b0442b470671ae6fc2892d2a74a90e335f87a7d6100ff55afa2e67a"}
Sep 16 12:12:16.924530 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:16.924105    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcefd37a5b0442b470671ae6fc2892d2a74a90e335f87a7d6100ff55afa2e67a"
Sep 16 12:12:16.924530 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:16.924115    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0qm87c"
Sep 16 12:12:18.934211 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:18.934128    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x" event={"ID":"c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4","Type":"ContainerStarted","Data":"b016d3ceecb3170dac3ee40ad1acde5d719ffb457cf388bfc7f1bdd50ea6c3c8"}
Sep 16 12:12:19.941212 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:19.941174    2579 generic.go:356] "Generic (PLEG): container finished" podID="c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4" containerID="b016d3ceecb3170dac3ee40ad1acde5d719ffb457cf388bfc7f1bdd50ea6c3c8" exitCode=0
Sep 16 12:12:19.941598 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:19.941262    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x" event={"ID":"c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4","Type":"ContainerDied","Data":"b016d3ceecb3170dac3ee40ad1acde5d719ffb457cf388bfc7f1bdd50ea6c3c8"}
Sep 16 12:12:20.948209 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:20.948169    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x" event={"ID":"c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4","Type":"ContainerStarted","Data":"064d536f0a5003a66291441844ad380be37f2205a0d2052a9c00f6fc9ef21228"}
Sep 16 12:12:20.948747 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:20.948271    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x"
Sep 16 12:12:20.972207 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:20.972148    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x" podStartSLOduration=2.71197582 podStartE2EDuration="6.972130602s" totalImagesPullingTime="3.209890893s" totalInitContainerRuntime="1.050263889s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:12:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:12:20.969566903 +0000 UTC m=+534.670638229" watchObservedRunningTime="2026-09-16 12:12:20.972130602 +0000 UTC m=+534.673201928"
Sep 16 12:12:23.898303 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:23.898270    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-74d55495b7-pcng2"
Sep 16 12:12:31.955146 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:31.955112    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6b5b57bf8b-2446x"
Sep 16 12:12:45.450306 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:45.450268    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-njsx5"]
Sep 16 12:12:45.450807 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:45.450599    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d84d6f7-9cf6-4696-8b77-ba70bf50c12b" containerName="extract"
Sep 16 12:12:45.450807 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:45.450610    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d84d6f7-9cf6-4696-8b77-ba70bf50c12b" containerName="extract"
Sep 16 12:12:45.450807 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:45.450628    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d84d6f7-9cf6-4696-8b77-ba70bf50c12b" containerName="util"
Sep 16 12:12:45.450807 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:45.450633    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d84d6f7-9cf6-4696-8b77-ba70bf50c12b" containerName="util"
Sep 16 12:12:45.450807 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:45.450644    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d84d6f7-9cf6-4696-8b77-ba70bf50c12b" containerName="pull"
Sep 16 12:12:45.450807 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:45.450650    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d84d6f7-9cf6-4696-8b77-ba70bf50c12b" containerName="pull"
Sep 16 12:12:45.450807 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:45.450734    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d84d6f7-9cf6-4696-8b77-ba70bf50c12b" containerName="extract"
Sep 16 12:12:45.455729 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:45.455711    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-njsx5"
Sep 16 12:12:45.458158 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:45.458136    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 16 12:12:45.465970 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:45.465946    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-njsx5"]
Sep 16 12:12:45.601712 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:45.601679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pzn4\" (UniqueName: \"kubernetes.io/projected/6aaabad9-9969-41bf-a8e3-620e6f1772f5-kube-api-access-4pzn4\") pod \"model-serving-api-54d89bd5fc-njsx5\" (UID: \"6aaabad9-9969-41bf-a8e3-620e6f1772f5\") " pod="opendatahub/model-serving-api-54d89bd5fc-njsx5"
Sep 16 12:12:45.601886 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:45.601803    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aaabad9-9969-41bf-a8e3-620e6f1772f5-tls-certs\") pod \"model-serving-api-54d89bd5fc-njsx5\" (UID: \"6aaabad9-9969-41bf-a8e3-620e6f1772f5\") " pod="opendatahub/model-serving-api-54d89bd5fc-njsx5"
Sep 16 12:12:45.702696 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:45.702575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aaabad9-9969-41bf-a8e3-620e6f1772f5-tls-certs\") pod \"model-serving-api-54d89bd5fc-njsx5\" (UID: \"6aaabad9-9969-41bf-a8e3-620e6f1772f5\") " pod="opendatahub/model-serving-api-54d89bd5fc-njsx5"
Sep 16 12:12:45.702696 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:45.702641    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pzn4\" (UniqueName: \"kubernetes.io/projected/6aaabad9-9969-41bf-a8e3-620e6f1772f5-kube-api-access-4pzn4\") pod \"model-serving-api-54d89bd5fc-njsx5\" (UID: \"6aaabad9-9969-41bf-a8e3-620e6f1772f5\") " pod="opendatahub/model-serving-api-54d89bd5fc-njsx5"
Sep 16 12:12:45.704992 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:45.704971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6aaabad9-9969-41bf-a8e3-620e6f1772f5-tls-certs\") pod \"model-serving-api-54d89bd5fc-njsx5\" (UID: \"6aaabad9-9969-41bf-a8e3-620e6f1772f5\") " pod="opendatahub/model-serving-api-54d89bd5fc-njsx5"
Sep 16 12:12:45.710271 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:45.710246    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pzn4\" (UniqueName: \"kubernetes.io/projected/6aaabad9-9969-41bf-a8e3-620e6f1772f5-kube-api-access-4pzn4\") pod \"model-serving-api-54d89bd5fc-njsx5\" (UID: \"6aaabad9-9969-41bf-a8e3-620e6f1772f5\") " pod="opendatahub/model-serving-api-54d89bd5fc-njsx5"
Sep 16 12:12:45.766976 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:45.766947    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-njsx5"
Sep 16 12:12:46.035630 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:46.035560    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-njsx5"]
Sep 16 12:12:46.038955 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:12:46.038928    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aaabad9_9969_41bf_a8e3_620e6f1772f5.slice/crio-10c09168ddffda1c08ac70a6c3ddba2fc79e3a7e05470a879d5aa5e9d818d92f WatchSource:0}: Error finding container 10c09168ddffda1c08ac70a6c3ddba2fc79e3a7e05470a879d5aa5e9d818d92f: Status 404 returned error can't find the container with id 10c09168ddffda1c08ac70a6c3ddba2fc79e3a7e05470a879d5aa5e9d818d92f
Sep 16 12:12:46.067434 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:46.067404    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-njsx5" event={"ID":"6aaabad9-9969-41bf-a8e3-620e6f1772f5","Type":"ContainerStarted","Data":"10c09168ddffda1c08ac70a6c3ddba2fc79e3a7e05470a879d5aa5e9d818d92f"}
Sep 16 12:12:49.085075 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:49.085028    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-njsx5" event={"ID":"6aaabad9-9969-41bf-a8e3-620e6f1772f5","Type":"ContainerStarted","Data":"76232a1a5466494488fd4151a19c35dae948a2780fa737ad81bd3a26cfb33f01"}
Sep 16 12:12:49.085511 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:49.085158    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-njsx5"
Sep 16 12:12:49.103076 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:49.103035    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-njsx5" podStartSLOduration=1.9712454 podStartE2EDuration="4.103020874s" totalImagesPullingTime="2.131775474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:12:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:12:49.101210322 +0000 UTC m=+562.802281648" watchObservedRunningTime="2026-09-16 12:12:49.103020874 +0000 UTC m=+562.804092587"
Sep 16 12:12:52.133094 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.133057    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"]
Sep 16 12:12:52.135505 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.135482    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.138576 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.138551    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 12:12:52.138736 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.138602    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 12:12:52.138736 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.138708    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-4fz6r\""
Sep 16 12:12:52.140482 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.140464    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 12:12:52.152985 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.152961    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.153079 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.152995    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.153079 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.153012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.153079 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.153037    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgv9s\" (UniqueName: \"kubernetes.io/projected/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-kube-api-access-wgv9s\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.153079 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.153054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.153079 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.153072    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.153325 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.153104    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.153325 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.153154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.153325 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.153247    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.154229 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.154207    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"]
Sep 16 12:12:52.254009 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.253970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.254175 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.254031    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.254175 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.254060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.254175 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.254077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.254175 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.254094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgv9s\" (UniqueName: \"kubernetes.io/projected/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-kube-api-access-wgv9s\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.254175 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.254114    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.254175 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.254139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.254175 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.254173    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.254614 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.254198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.254614 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.254408    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.254614 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.254504    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.254614 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.254591    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.254819 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.254628    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.255009 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.254990    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.256540 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.256518    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.256627 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.256572    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.264403 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.264370    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.265046 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.265024    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgv9s\" (UniqueName: \"kubernetes.io/projected/54b8d903-4ad2-41f5-8db2-a22e9c8e76ca-kube-api-access-wgv9s\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp\" (UID: \"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.446848 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.446755    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:52.748225 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:52.748150    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"]
Sep 16 12:12:52.751182 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:12:52.751155    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b8d903_4ad2_41f5_8db2_a22e9c8e76ca.slice/crio-98698f8d56f0895c0cb7dea10cdd005877336bac831f566c79497d82349fe5bc WatchSource:0}: Error finding container 98698f8d56f0895c0cb7dea10cdd005877336bac831f566c79497d82349fe5bc: Status 404 returned error can't find the container with id 98698f8d56f0895c0cb7dea10cdd005877336bac831f566c79497d82349fe5bc
Sep 16 12:12:53.110931 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:53.110894    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp" event={"ID":"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca","Type":"ContainerStarted","Data":"98698f8d56f0895c0cb7dea10cdd005877336bac831f566c79497d82349fe5bc"}
Sep 16 12:12:54.695793 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:54.695759    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-n5f8s"]
Sep 16 12:12:54.700993 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:54.700969    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5f8s"
Sep 16 12:12:54.703493 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:54.703470    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 16 12:12:54.703616 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:54.703490    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-8qgfd\""
Sep 16 12:12:54.710000 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:54.709940    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-n5f8s"]
Sep 16 12:12:54.715542 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:54.715520    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-gwhkn"]
Sep 16 12:12:54.717998 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:54.717975    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwhkn"
Sep 16 12:12:54.720317 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:54.720169    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 16 12:12:54.720317 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:54.720190    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-7xdnj\""
Sep 16 12:12:54.728369 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:54.728348    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-gwhkn"]
Sep 16 12:12:54.780954 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:54.780919    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b325748-891a-4d74-9bd1-ccbc177d05f2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gwhkn\" (UID: \"2b325748-891a-4d74-9bd1-ccbc177d05f2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwhkn"
Sep 16 12:12:54.781122 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:54.780965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96g4g\" (UniqueName: \"kubernetes.io/projected/2b325748-891a-4d74-9bd1-ccbc177d05f2-kube-api-access-96g4g\") pod \"llmisvc-controller-manager-777c6bd7d5-gwhkn\" (UID: \"2b325748-891a-4d74-9bd1-ccbc177d05f2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwhkn"
Sep 16 12:12:54.781182 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:54.781118    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntl9s\" (UniqueName: \"kubernetes.io/projected/f7b110d0-4396-476b-a69b-91f00e2405cf-kube-api-access-ntl9s\") pod \"kserve-controller-manager-d6f4b65b8-n5f8s\" (UID: \"f7b110d0-4396-476b-a69b-91f00e2405cf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5f8s"
Sep 16 12:12:54.781182 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:54.781169    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7b110d0-4396-476b-a69b-91f00e2405cf-cert\") pod \"kserve-controller-manager-d6f4b65b8-n5f8s\" (UID: \"f7b110d0-4396-476b-a69b-91f00e2405cf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5f8s"
Sep 16 12:12:54.881797 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:54.881762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7b110d0-4396-476b-a69b-91f00e2405cf-cert\") pod \"kserve-controller-manager-d6f4b65b8-n5f8s\" (UID: \"f7b110d0-4396-476b-a69b-91f00e2405cf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5f8s"
Sep 16 12:12:54.881985 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:54.881832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b325748-891a-4d74-9bd1-ccbc177d05f2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gwhkn\" (UID: \"2b325748-891a-4d74-9bd1-ccbc177d05f2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwhkn"
Sep 16 12:12:54.881985 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:54.881861    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96g4g\" (UniqueName: \"kubernetes.io/projected/2b325748-891a-4d74-9bd1-ccbc177d05f2-kube-api-access-96g4g\") pod \"llmisvc-controller-manager-777c6bd7d5-gwhkn\" (UID: \"2b325748-891a-4d74-9bd1-ccbc177d05f2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwhkn"
Sep 16 12:12:54.881985 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:54.881918    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntl9s\" (UniqueName: \"kubernetes.io/projected/f7b110d0-4396-476b-a69b-91f00e2405cf-kube-api-access-ntl9s\") pod \"kserve-controller-manager-d6f4b65b8-n5f8s\" (UID: \"f7b110d0-4396-476b-a69b-91f00e2405cf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5f8s"
Sep 16 12:12:54.881985 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:12:54.881933    2579 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 16 12:12:54.882218 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:12:54.882008    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7b110d0-4396-476b-a69b-91f00e2405cf-cert podName:f7b110d0-4396-476b-a69b-91f00e2405cf nodeName:}" failed. No retries permitted until 2026-09-16 12:12:55.38198652 +0000 UTC m=+569.083057836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f7b110d0-4396-476b-a69b-91f00e2405cf-cert") pod "kserve-controller-manager-d6f4b65b8-n5f8s" (UID: "f7b110d0-4396-476b-a69b-91f00e2405cf") : secret "kserve-webhook-server-cert" not found
Sep 16 12:12:54.884782 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:54.884756    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b325748-891a-4d74-9bd1-ccbc177d05f2-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gwhkn\" (UID: \"2b325748-891a-4d74-9bd1-ccbc177d05f2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwhkn"
Sep 16 12:12:54.895437 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:54.895415    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96g4g\" (UniqueName: \"kubernetes.io/projected/2b325748-891a-4d74-9bd1-ccbc177d05f2-kube-api-access-96g4g\") pod \"llmisvc-controller-manager-777c6bd7d5-gwhkn\" (UID: \"2b325748-891a-4d74-9bd1-ccbc177d05f2\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwhkn"
Sep 16 12:12:54.895688 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:54.895640    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntl9s\" (UniqueName: \"kubernetes.io/projected/f7b110d0-4396-476b-a69b-91f00e2405cf-kube-api-access-ntl9s\") pod \"kserve-controller-manager-d6f4b65b8-n5f8s\" (UID: \"f7b110d0-4396-476b-a69b-91f00e2405cf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5f8s"
Sep 16 12:12:55.030722 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:55.030592    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwhkn"
Sep 16 12:12:55.376243 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:55.376192    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:12:55.376446 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:55.376420    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:12:55.376524 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:55.376476    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:12:55.386268 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:55.386228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7b110d0-4396-476b-a69b-91f00e2405cf-cert\") pod \"kserve-controller-manager-d6f4b65b8-n5f8s\" (UID: \"f7b110d0-4396-476b-a69b-91f00e2405cf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5f8s"
Sep 16 12:12:55.388532 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:55.388510    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7b110d0-4396-476b-a69b-91f00e2405cf-cert\") pod \"kserve-controller-manager-d6f4b65b8-n5f8s\" (UID: \"f7b110d0-4396-476b-a69b-91f00e2405cf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5f8s"
Sep 16 12:12:55.609138 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:55.609105    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-gwhkn"]
Sep 16 12:12:55.610754 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:12:55.610718    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2b325748_891a_4d74_9bd1_ccbc177d05f2.slice/crio-f073e8b991061dec32ffa8a0312cda0bb34084a6b3364e7f84c18d46c924d791 WatchSource:0}: Error finding container f073e8b991061dec32ffa8a0312cda0bb34084a6b3364e7f84c18d46c924d791: Status 404 returned error can't find the container with id f073e8b991061dec32ffa8a0312cda0bb34084a6b3364e7f84c18d46c924d791
Sep 16 12:12:55.613998 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:55.613938    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5f8s"
Sep 16 12:12:55.902739 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:55.902648    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-n5f8s"]
Sep 16 12:12:55.906015 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:12:55.905988    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b110d0_4396_476b_a69b_91f00e2405cf.slice/crio-e2fc262b75602c81bc0532de90fdf923a37fd6178acd98a88ca0858174101044 WatchSource:0}: Error finding container e2fc262b75602c81bc0532de90fdf923a37fd6178acd98a88ca0858174101044: Status 404 returned error can't find the container with id e2fc262b75602c81bc0532de90fdf923a37fd6178acd98a88ca0858174101044
Sep 16 12:12:56.127706 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:56.127670    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5f8s" event={"ID":"f7b110d0-4396-476b-a69b-91f00e2405cf","Type":"ContainerStarted","Data":"e2fc262b75602c81bc0532de90fdf923a37fd6178acd98a88ca0858174101044"}
Sep 16 12:12:56.128896 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:56.128866    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwhkn" event={"ID":"2b325748-891a-4d74-9bd1-ccbc177d05f2","Type":"ContainerStarted","Data":"f073e8b991061dec32ffa8a0312cda0bb34084a6b3364e7f84c18d46c924d791"}
Sep 16 12:12:56.130458 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:56.130430    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp" event={"ID":"54b8d903-4ad2-41f5-8db2-a22e9c8e76ca","Type":"ContainerStarted","Data":"2af5b86a7b56ff1163662592d31d30598f830c110856e8b681d98e1e978d9574"}
Sep 16 12:12:56.152406 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:56.152351    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp" podStartSLOduration=1.529694568 podStartE2EDuration="4.152336683s" totalImagesPullingTime="2.622642115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:12:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:12:56.150442419 +0000 UTC m=+569.851513746" watchObservedRunningTime="2026-09-16 12:12:56.152336683 +0000 UTC m=+569.853408008"
Sep 16 12:12:56.447878 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:56.447838    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:56.453636 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:56.453598    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:57.135751 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:57.135715    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:57.136824 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:57.136802    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp"
Sep 16 12:12:59.149063 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:59.149021    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5f8s" event={"ID":"f7b110d0-4396-476b-a69b-91f00e2405cf","Type":"ContainerStarted","Data":"8225abaceacf18c2184d27e8b3605bdda5552590c68720a7952720593a1b1651"}
Sep 16 12:12:59.149521 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:59.149099    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5f8s"
Sep 16 12:12:59.150917 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:59.150888    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwhkn" event={"ID":"2b325748-891a-4d74-9bd1-ccbc177d05f2","Type":"ContainerStarted","Data":"0f713f7d2e77aefa8325416d3b510f2f480a1f9e5ae3bf6d3e708cf451c1f3d7"}
Sep 16 12:12:59.151050 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:59.150974    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwhkn"
Sep 16 12:12:59.164839 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:59.164795    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5f8s" podStartSLOduration=2.392966289 podStartE2EDuration="5.164781802s" totalImagesPullingTime="2.771815513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:12:59.163791283 +0000 UTC m=+572.864862610" watchObservedRunningTime="2026-09-16 12:12:59.164781802 +0000 UTC m=+572.865853127"
Sep 16 12:12:59.179866 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:12:59.179821    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwhkn" podStartSLOduration=2.113313189 podStartE2EDuration="5.179802722s" totalImagesPullingTime="3.066489533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:12:59.17942818 +0000 UTC m=+572.880499500" watchObservedRunningTime="2026-09-16 12:12:59.179802722 +0000 UTC m=+572.880874049"
Sep 16 12:13:00.094003 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:00.093972    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-njsx5"
Sep 16 12:13:30.158313 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:30.158270    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gwhkn"
Sep 16 12:13:30.161473 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:30.161451    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-n5f8s"
Sep 16 12:13:49.204620 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.204544    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd"]
Sep 16 12:13:49.210798 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.210779    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd"
Sep 16 12:13:49.214631 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.214511    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r4zt6\""
Sep 16 12:13:49.214631 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.214515    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 12:13:49.214828 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.214735    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 12:13:49.215524 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.215504    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd"]
Sep 16 12:13:49.244131 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.244100    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gwl5\" (UniqueName: \"kubernetes.io/projected/b7635b03-5ad7-46f5-b584-7c2439e4b196-kube-api-access-7gwl5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd\" (UID: \"b7635b03-5ad7-46f5-b584-7c2439e4b196\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd"
Sep 16 12:13:49.244256 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.244154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7635b03-5ad7-46f5-b584-7c2439e4b196-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd\" (UID: \"b7635b03-5ad7-46f5-b584-7c2439e4b196\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd"
Sep 16 12:13:49.244256 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.244206    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7635b03-5ad7-46f5-b584-7c2439e4b196-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd\" (UID: \"b7635b03-5ad7-46f5-b584-7c2439e4b196\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd"
Sep 16 12:13:49.307346 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.307310    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q"]
Sep 16 12:13:49.309880 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.309864    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q"
Sep 16 12:13:49.318136 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.318112    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q"]
Sep 16 12:13:49.344944 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.344905    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gwl5\" (UniqueName: \"kubernetes.io/projected/b7635b03-5ad7-46f5-b584-7c2439e4b196-kube-api-access-7gwl5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd\" (UID: \"b7635b03-5ad7-46f5-b584-7c2439e4b196\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd"
Sep 16 12:13:49.345121 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.344984    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q\" (UID: \"d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q"
Sep 16 12:13:49.345121 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.345014    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8wg8\" (UniqueName: \"kubernetes.io/projected/d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4-kube-api-access-x8wg8\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q\" (UID: \"d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q"
Sep 16 12:13:49.345121 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.345077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7635b03-5ad7-46f5-b584-7c2439e4b196-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd\" (UID: \"b7635b03-5ad7-46f5-b584-7c2439e4b196\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd"
Sep 16 12:13:49.345121 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.345118    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q\" (UID: \"d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q"
Sep 16 12:13:49.345295 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.345151    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7635b03-5ad7-46f5-b584-7c2439e4b196-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd\" (UID: \"b7635b03-5ad7-46f5-b584-7c2439e4b196\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd"
Sep 16 12:13:49.345463 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.345447    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7635b03-5ad7-46f5-b584-7c2439e4b196-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd\" (UID: \"b7635b03-5ad7-46f5-b584-7c2439e4b196\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd"
Sep 16 12:13:49.345500 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.345474    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7635b03-5ad7-46f5-b584-7c2439e4b196-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd\" (UID: \"b7635b03-5ad7-46f5-b584-7c2439e4b196\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd"
Sep 16 12:13:49.352922 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.352894    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gwl5\" (UniqueName: \"kubernetes.io/projected/b7635b03-5ad7-46f5-b584-7c2439e4b196-kube-api-access-7gwl5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd\" (UID: \"b7635b03-5ad7-46f5-b584-7c2439e4b196\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd"
Sep 16 12:13:49.406028 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.405960    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2"]
Sep 16 12:13:49.408748 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.408732    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2"
Sep 16 12:13:49.418630 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.417535    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2"]
Sep 16 12:13:49.446192 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.446161    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3c47389-88fd-4804-bf48-2bb391470716-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2\" (UID: \"d3c47389-88fd-4804-bf48-2bb391470716\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2"
Sep 16 12:13:49.446192 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.446196    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ch8v\" (UniqueName: \"kubernetes.io/projected/d3c47389-88fd-4804-bf48-2bb391470716-kube-api-access-9ch8v\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2\" (UID: \"d3c47389-88fd-4804-bf48-2bb391470716\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2"
Sep 16 12:13:49.446391 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.446297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q\" (UID: \"d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q"
Sep 16 12:13:49.446391 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.446323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8wg8\" (UniqueName: \"kubernetes.io/projected/d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4-kube-api-access-x8wg8\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q\" (UID: \"d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q"
Sep 16 12:13:49.446391 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.446343    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3c47389-88fd-4804-bf48-2bb391470716-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2\" (UID: \"d3c47389-88fd-4804-bf48-2bb391470716\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2"
Sep 16 12:13:49.446391 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.446374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q\" (UID: \"d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q"
Sep 16 12:13:49.446711 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.446681    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q\" (UID: \"d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q"
Sep 16 12:13:49.446819 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.446737    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q\" (UID: \"d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q"
Sep 16 12:13:49.454331 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.454312    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8wg8\" (UniqueName: \"kubernetes.io/projected/d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4-kube-api-access-x8wg8\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q\" (UID: \"d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q"
Sep 16 12:13:49.511490 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.511416    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9"]
Sep 16 12:13:49.514117 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.514100    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9"
Sep 16 12:13:49.521440 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.521209    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd"
Sep 16 12:13:49.522094 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.522073    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9"]
Sep 16 12:13:49.547402 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.547373    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r65th\" (UniqueName: \"kubernetes.io/projected/5bcd29ef-5a04-4005-84a1-a9fbc8d4b391-kube-api-access-r65th\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9\" (UID: \"5bcd29ef-5a04-4005-84a1-a9fbc8d4b391\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9"
Sep 16 12:13:49.547547 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.547422    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3c47389-88fd-4804-bf48-2bb391470716-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2\" (UID: \"d3c47389-88fd-4804-bf48-2bb391470716\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2"
Sep 16 12:13:49.547547 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.547444    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ch8v\" (UniqueName: \"kubernetes.io/projected/d3c47389-88fd-4804-bf48-2bb391470716-kube-api-access-9ch8v\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2\" (UID: \"d3c47389-88fd-4804-bf48-2bb391470716\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2"
Sep 16 12:13:49.547547 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.547512    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5bcd29ef-5a04-4005-84a1-a9fbc8d4b391-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9\" (UID: \"5bcd29ef-5a04-4005-84a1-a9fbc8d4b391\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9"
Sep 16 12:13:49.547694 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.547569    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5bcd29ef-5a04-4005-84a1-a9fbc8d4b391-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9\" (UID: \"5bcd29ef-5a04-4005-84a1-a9fbc8d4b391\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9"
Sep 16 12:13:49.547694 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.547641    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3c47389-88fd-4804-bf48-2bb391470716-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2\" (UID: \"d3c47389-88fd-4804-bf48-2bb391470716\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2"
Sep 16 12:13:49.547867 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.547849    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3c47389-88fd-4804-bf48-2bb391470716-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2\" (UID: \"d3c47389-88fd-4804-bf48-2bb391470716\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2"
Sep 16 12:13:49.547955 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.547939    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3c47389-88fd-4804-bf48-2bb391470716-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2\" (UID: \"d3c47389-88fd-4804-bf48-2bb391470716\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2"
Sep 16 12:13:49.557222 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.557194    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ch8v\" (UniqueName: \"kubernetes.io/projected/d3c47389-88fd-4804-bf48-2bb391470716-kube-api-access-9ch8v\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2\" (UID: \"d3c47389-88fd-4804-bf48-2bb391470716\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2"
Sep 16 12:13:49.618924 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.618892    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q"
Sep 16 12:13:49.648031 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.647999    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r65th\" (UniqueName: \"kubernetes.io/projected/5bcd29ef-5a04-4005-84a1-a9fbc8d4b391-kube-api-access-r65th\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9\" (UID: \"5bcd29ef-5a04-4005-84a1-a9fbc8d4b391\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9"
Sep 16 12:13:49.648170 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.648060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5bcd29ef-5a04-4005-84a1-a9fbc8d4b391-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9\" (UID: \"5bcd29ef-5a04-4005-84a1-a9fbc8d4b391\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9"
Sep 16 12:13:49.648170 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.648081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5bcd29ef-5a04-4005-84a1-a9fbc8d4b391-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9\" (UID: \"5bcd29ef-5a04-4005-84a1-a9fbc8d4b391\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9"
Sep 16 12:13:49.648398 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.648381    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5bcd29ef-5a04-4005-84a1-a9fbc8d4b391-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9\" (UID: \"5bcd29ef-5a04-4005-84a1-a9fbc8d4b391\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9"
Sep 16 12:13:49.648471 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.648431    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5bcd29ef-5a04-4005-84a1-a9fbc8d4b391-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9\" (UID: \"5bcd29ef-5a04-4005-84a1-a9fbc8d4b391\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9"
Sep 16 12:13:49.657828 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.657805    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r65th\" (UniqueName: \"kubernetes.io/projected/5bcd29ef-5a04-4005-84a1-a9fbc8d4b391-kube-api-access-r65th\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9\" (UID: \"5bcd29ef-5a04-4005-84a1-a9fbc8d4b391\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9"
Sep 16 12:13:49.719630 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.719597    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2"
Sep 16 12:13:49.799453 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.799421    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd"]
Sep 16 12:13:49.800828 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:13:49.800804    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7635b03_5ad7_46f5_b584_7c2439e4b196.slice/crio-f76fddd229797cc0f33a0f8782a93bccba2a38d172c691deaeea2bdf6ca99eca WatchSource:0}: Error finding container f76fddd229797cc0f33a0f8782a93bccba2a38d172c691deaeea2bdf6ca99eca: Status 404 returned error can't find the container with id f76fddd229797cc0f33a0f8782a93bccba2a38d172c691deaeea2bdf6ca99eca
Sep 16 12:13:49.824751 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.824728    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9"
Sep 16 12:13:49.890650 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.890623    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q"]
Sep 16 12:13:49.893072 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:13:49.893046    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5dfa84f_bd71_4686_87c3_7d0b8dcbcde4.slice/crio-f908444619b4ae22a62867a18fbf8fe02d89d3aa95ea68374dacf67270e5b2c9 WatchSource:0}: Error finding container f908444619b4ae22a62867a18fbf8fe02d89d3aa95ea68374dacf67270e5b2c9: Status 404 returned error can't find the container with id f908444619b4ae22a62867a18fbf8fe02d89d3aa95ea68374dacf67270e5b2c9
Sep 16 12:13:49.987868 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:49.987844    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2"]
Sep 16 12:13:49.989259 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:13:49.989229    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c47389_88fd_4804_bf48_2bb391470716.slice/crio-1d5d2b5a903f7e79f311384af9380e936ab5e1a5172239531f88839e0cef5722 WatchSource:0}: Error finding container 1d5d2b5a903f7e79f311384af9380e936ab5e1a5172239531f88839e0cef5722: Status 404 returned error can't find the container with id 1d5d2b5a903f7e79f311384af9380e936ab5e1a5172239531f88839e0cef5722
Sep 16 12:13:50.098925 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:50.098901    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9"]
Sep 16 12:13:50.137502 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:13:50.137473    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bcd29ef_5a04_4005_84a1_a9fbc8d4b391.slice/crio-62675df1d3ce916305b0646d6987272aae58e9d81b010f84733ca7d0412c4863 WatchSource:0}: Error finding container 62675df1d3ce916305b0646d6987272aae58e9d81b010f84733ca7d0412c4863: Status 404 returned error can't find the container with id 62675df1d3ce916305b0646d6987272aae58e9d81b010f84733ca7d0412c4863
Sep 16 12:13:50.404427 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:50.404393    2579 generic.go:356] "Generic (PLEG): container finished" podID="d3c47389-88fd-4804-bf48-2bb391470716" containerID="4efa0774835e39658fa92ffcd397b568050b86803f17905ad6617e22f57d9245" exitCode=0
Sep 16 12:13:50.404877 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:50.404482    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2" event={"ID":"d3c47389-88fd-4804-bf48-2bb391470716","Type":"ContainerDied","Data":"4efa0774835e39658fa92ffcd397b568050b86803f17905ad6617e22f57d9245"}
Sep 16 12:13:50.404877 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:50.404536    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2" event={"ID":"d3c47389-88fd-4804-bf48-2bb391470716","Type":"ContainerStarted","Data":"1d5d2b5a903f7e79f311384af9380e936ab5e1a5172239531f88839e0cef5722"}
Sep 16 12:13:50.405640 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:50.405622    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:13:50.406220 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:50.406200    2579 generic.go:356] "Generic (PLEG): container finished" podID="b7635b03-5ad7-46f5-b584-7c2439e4b196" containerID="5bea1a063d57970812f0b9182adfae73dad8fe3ff734aa2a2904079cb22834be" exitCode=0
Sep 16 12:13:50.406306 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:50.406282    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd" event={"ID":"b7635b03-5ad7-46f5-b584-7c2439e4b196","Type":"ContainerDied","Data":"5bea1a063d57970812f0b9182adfae73dad8fe3ff734aa2a2904079cb22834be"}
Sep 16 12:13:50.406361 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:50.406334    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd" event={"ID":"b7635b03-5ad7-46f5-b584-7c2439e4b196","Type":"ContainerStarted","Data":"f76fddd229797cc0f33a0f8782a93bccba2a38d172c691deaeea2bdf6ca99eca"}
Sep 16 12:13:50.408045 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:50.408025    2579 generic.go:356] "Generic (PLEG): container finished" podID="5bcd29ef-5a04-4005-84a1-a9fbc8d4b391" containerID="43e4ef72d7ef978e9f2bde549aa21e712c038bdc0627ca1581e3ff32b0f73a67" exitCode=0
Sep 16 12:13:50.408128 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:50.408078    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9" event={"ID":"5bcd29ef-5a04-4005-84a1-a9fbc8d4b391","Type":"ContainerDied","Data":"43e4ef72d7ef978e9f2bde549aa21e712c038bdc0627ca1581e3ff32b0f73a67"}
Sep 16 12:13:50.408191 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:50.408141    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9" event={"ID":"5bcd29ef-5a04-4005-84a1-a9fbc8d4b391","Type":"ContainerStarted","Data":"62675df1d3ce916305b0646d6987272aae58e9d81b010f84733ca7d0412c4863"}
Sep 16 12:13:50.409874 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:50.409856    2579 generic.go:356] "Generic (PLEG): container finished" podID="d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4" containerID="c5b3669dd61537c7fcfe8cc715c071d54b644c115ca928a7395cb34b3aaa4776" exitCode=0
Sep 16 12:13:50.409956 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:50.409913    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q" event={"ID":"d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4","Type":"ContainerDied","Data":"c5b3669dd61537c7fcfe8cc715c071d54b644c115ca928a7395cb34b3aaa4776"}
Sep 16 12:13:50.409956 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:50.409938    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q" event={"ID":"d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4","Type":"ContainerStarted","Data":"f908444619b4ae22a62867a18fbf8fe02d89d3aa95ea68374dacf67270e5b2c9"}
Sep 16 12:13:51.417291 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:51.417263    2579 generic.go:356] "Generic (PLEG): container finished" podID="d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4" containerID="05f8c33bffdd889564b8068b2ff696492972c708c13a192e48525c3754aaf115" exitCode=0
Sep 16 12:13:51.417641 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:51.417345    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q" event={"ID":"d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4","Type":"ContainerDied","Data":"05f8c33bffdd889564b8068b2ff696492972c708c13a192e48525c3754aaf115"}
Sep 16 12:13:51.419423 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:51.419399    2579 generic.go:356] "Generic (PLEG): container finished" podID="d3c47389-88fd-4804-bf48-2bb391470716" containerID="5c06ff241ea35bdd856f9886a1a078c3bf0d5759131833900e30a194ddc3de2e" exitCode=0
Sep 16 12:13:51.419515 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:51.419473    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2" event={"ID":"d3c47389-88fd-4804-bf48-2bb391470716","Type":"ContainerDied","Data":"5c06ff241ea35bdd856f9886a1a078c3bf0d5759131833900e30a194ddc3de2e"}
Sep 16 12:13:51.421531 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:51.421502    2579 generic.go:356] "Generic (PLEG): container finished" podID="b7635b03-5ad7-46f5-b584-7c2439e4b196" containerID="e8714c457787e15583f11c6988352e201572177a9374e528a3e9fb3a80391cd0" exitCode=0
Sep 16 12:13:51.421601 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:51.421549    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd" event={"ID":"b7635b03-5ad7-46f5-b584-7c2439e4b196","Type":"ContainerDied","Data":"e8714c457787e15583f11c6988352e201572177a9374e528a3e9fb3a80391cd0"}
Sep 16 12:13:52.429522 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:52.429484    2579 generic.go:356] "Generic (PLEG): container finished" podID="d3c47389-88fd-4804-bf48-2bb391470716" containerID="d402167e7749bccef803e20bb4588c83c0e80a1cb713aa5701c51d57107925bb" exitCode=0
Sep 16 12:13:52.429964 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:52.429562    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2" event={"ID":"d3c47389-88fd-4804-bf48-2bb391470716","Type":"ContainerDied","Data":"d402167e7749bccef803e20bb4588c83c0e80a1cb713aa5701c51d57107925bb"}
Sep 16 12:13:52.431650 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:52.431621    2579 generic.go:356] "Generic (PLEG): container finished" podID="b7635b03-5ad7-46f5-b584-7c2439e4b196" containerID="c971b6fe1fbeafd5e0176c5377860afeb78a5d4bd96a56a6ca22a2a2e1bdd058" exitCode=0
Sep 16 12:13:52.431784 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:52.431714    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd" event={"ID":"b7635b03-5ad7-46f5-b584-7c2439e4b196","Type":"ContainerDied","Data":"c971b6fe1fbeafd5e0176c5377860afeb78a5d4bd96a56a6ca22a2a2e1bdd058"}
Sep 16 12:13:52.433474 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:52.433453    2579 generic.go:356] "Generic (PLEG): container finished" podID="5bcd29ef-5a04-4005-84a1-a9fbc8d4b391" containerID="bd84721ff2e5d8f69bc18b1b4a317402016e76c525ed8280db213a90119e9915" exitCode=0
Sep 16 12:13:52.433560 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:52.433534    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9" event={"ID":"5bcd29ef-5a04-4005-84a1-a9fbc8d4b391","Type":"ContainerDied","Data":"bd84721ff2e5d8f69bc18b1b4a317402016e76c525ed8280db213a90119e9915"}
Sep 16 12:13:52.435826 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:52.435806    2579 generic.go:356] "Generic (PLEG): container finished" podID="d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4" containerID="b3645e4bff7eb78c642c0566f838b6a95a555ffe35d5c1df5f35038d9fccfeab" exitCode=0
Sep 16 12:13:52.435910 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:52.435839    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q" event={"ID":"d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4","Type":"ContainerDied","Data":"b3645e4bff7eb78c642c0566f838b6a95a555ffe35d5c1df5f35038d9fccfeab"}
Sep 16 12:13:53.442946 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.442908    2579 generic.go:356] "Generic (PLEG): container finished" podID="5bcd29ef-5a04-4005-84a1-a9fbc8d4b391" containerID="b802ffcf4276964995c14ff4c29de4a124f3dfdb62b4efe911b55454255ed26c" exitCode=0
Sep 16 12:13:53.443417 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.442992    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9" event={"ID":"5bcd29ef-5a04-4005-84a1-a9fbc8d4b391","Type":"ContainerDied","Data":"b802ffcf4276964995c14ff4c29de4a124f3dfdb62b4efe911b55454255ed26c"}
Sep 16 12:13:53.592637 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.592605    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2"
Sep 16 12:13:53.636644 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.636611    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q"
Sep 16 12:13:53.639293 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.639274    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd"
Sep 16 12:13:53.678642 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.678613    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7635b03-5ad7-46f5-b584-7c2439e4b196-bundle\") pod \"b7635b03-5ad7-46f5-b584-7c2439e4b196\" (UID: \"b7635b03-5ad7-46f5-b584-7c2439e4b196\") "
Sep 16 12:13:53.678836 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.678676    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ch8v\" (UniqueName: \"kubernetes.io/projected/d3c47389-88fd-4804-bf48-2bb391470716-kube-api-access-9ch8v\") pod \"d3c47389-88fd-4804-bf48-2bb391470716\" (UID: \"d3c47389-88fd-4804-bf48-2bb391470716\") "
Sep 16 12:13:53.678836 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.678694    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3c47389-88fd-4804-bf48-2bb391470716-bundle\") pod \"d3c47389-88fd-4804-bf48-2bb391470716\" (UID: \"d3c47389-88fd-4804-bf48-2bb391470716\") "
Sep 16 12:13:53.678836 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.678726    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4-bundle\") pod \"d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4\" (UID: \"d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4\") "
Sep 16 12:13:53.678836 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.678753    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4-util\") pod \"d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4\" (UID: \"d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4\") "
Sep 16 12:13:53.678836 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.678788    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gwl5\" (UniqueName: \"kubernetes.io/projected/b7635b03-5ad7-46f5-b584-7c2439e4b196-kube-api-access-7gwl5\") pod \"b7635b03-5ad7-46f5-b584-7c2439e4b196\" (UID: \"b7635b03-5ad7-46f5-b584-7c2439e4b196\") "
Sep 16 12:13:53.678836 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.678812    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8wg8\" (UniqueName: \"kubernetes.io/projected/d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4-kube-api-access-x8wg8\") pod \"d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4\" (UID: \"d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4\") "
Sep 16 12:13:53.679136 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.678854    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3c47389-88fd-4804-bf48-2bb391470716-util\") pod \"d3c47389-88fd-4804-bf48-2bb391470716\" (UID: \"d3c47389-88fd-4804-bf48-2bb391470716\") "
Sep 16 12:13:53.679136 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.678907    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7635b03-5ad7-46f5-b584-7c2439e4b196-util\") pod \"b7635b03-5ad7-46f5-b584-7c2439e4b196\" (UID: \"b7635b03-5ad7-46f5-b584-7c2439e4b196\") "
Sep 16 12:13:53.679692 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.679421    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c47389-88fd-4804-bf48-2bb391470716-bundle" (OuterVolumeSpecName: "bundle") pod "d3c47389-88fd-4804-bf48-2bb391470716" (UID: "d3c47389-88fd-4804-bf48-2bb391470716"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:13:53.679692 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.679629    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7635b03-5ad7-46f5-b584-7c2439e4b196-bundle" (OuterVolumeSpecName: "bundle") pod "b7635b03-5ad7-46f5-b584-7c2439e4b196" (UID: "b7635b03-5ad7-46f5-b584-7c2439e4b196"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:13:53.680135 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.680112    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4-bundle" (OuterVolumeSpecName: "bundle") pod "d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4" (UID: "d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:13:53.681566 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.681536    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c47389-88fd-4804-bf48-2bb391470716-kube-api-access-9ch8v" (OuterVolumeSpecName: "kube-api-access-9ch8v") pod "d3c47389-88fd-4804-bf48-2bb391470716" (UID: "d3c47389-88fd-4804-bf48-2bb391470716"). InnerVolumeSpecName "kube-api-access-9ch8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:13:53.682250 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.682226    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7635b03-5ad7-46f5-b584-7c2439e4b196-kube-api-access-7gwl5" (OuterVolumeSpecName: "kube-api-access-7gwl5") pod "b7635b03-5ad7-46f5-b584-7c2439e4b196" (UID: "b7635b03-5ad7-46f5-b584-7c2439e4b196"). InnerVolumeSpecName "kube-api-access-7gwl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:13:53.682546 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.682511    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4-kube-api-access-x8wg8" (OuterVolumeSpecName: "kube-api-access-x8wg8") pod "d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4" (UID: "d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4"). InnerVolumeSpecName "kube-api-access-x8wg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:13:53.685057 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.685015    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4-util" (OuterVolumeSpecName: "util") pod "d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4" (UID: "d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:13:53.685855 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.685834    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c47389-88fd-4804-bf48-2bb391470716-util" (OuterVolumeSpecName: "util") pod "d3c47389-88fd-4804-bf48-2bb391470716" (UID: "d3c47389-88fd-4804-bf48-2bb391470716"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:13:53.686083 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.686065    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7635b03-5ad7-46f5-b584-7c2439e4b196-util" (OuterVolumeSpecName: "util") pod "b7635b03-5ad7-46f5-b584-7c2439e4b196" (UID: "b7635b03-5ad7-46f5-b584-7c2439e4b196"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:13:53.780068 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.779979    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7635b03-5ad7-46f5-b584-7c2439e4b196-util\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:13:53.780068 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.780011    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7635b03-5ad7-46f5-b584-7c2439e4b196-bundle\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:13:53.780068 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.780020    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ch8v\" (UniqueName: \"kubernetes.io/projected/d3c47389-88fd-4804-bf48-2bb391470716-kube-api-access-9ch8v\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:13:53.780068 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.780031    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d3c47389-88fd-4804-bf48-2bb391470716-bundle\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:13:53.780068 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.780039    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4-bundle\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:13:53.780068 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.780047    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4-util\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:13:53.780068 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.780057    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gwl5\" (UniqueName: \"kubernetes.io/projected/b7635b03-5ad7-46f5-b584-7c2439e4b196-kube-api-access-7gwl5\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:13:53.780068 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.780065    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8wg8\" (UniqueName: \"kubernetes.io/projected/d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4-kube-api-access-x8wg8\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:13:53.780068 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:53.780073    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d3c47389-88fd-4804-bf48-2bb391470716-util\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:13:54.450275 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:54.450243    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q"
Sep 16 12:13:54.450689 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:54.450273    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ff2q" event={"ID":"d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4","Type":"ContainerDied","Data":"f908444619b4ae22a62867a18fbf8fe02d89d3aa95ea68374dacf67270e5b2c9"}
Sep 16 12:13:54.450689 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:54.450327    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f908444619b4ae22a62867a18fbf8fe02d89d3aa95ea68374dacf67270e5b2c9"
Sep 16 12:13:54.452256 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:54.452237    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2"
Sep 16 12:13:54.452370 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:54.452233    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xn8h2" event={"ID":"d3c47389-88fd-4804-bf48-2bb391470716","Type":"ContainerDied","Data":"1d5d2b5a903f7e79f311384af9380e936ab5e1a5172239531f88839e0cef5722"}
Sep 16 12:13:54.452370 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:54.452352    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d5d2b5a903f7e79f311384af9380e936ab5e1a5172239531f88839e0cef5722"
Sep 16 12:13:54.454305 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:54.454287    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd"
Sep 16 12:13:54.454412 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:54.454304    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c57pnd" event={"ID":"b7635b03-5ad7-46f5-b584-7c2439e4b196","Type":"ContainerDied","Data":"f76fddd229797cc0f33a0f8782a93bccba2a38d172c691deaeea2bdf6ca99eca"}
Sep 16 12:13:54.454412 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:54.454337    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f76fddd229797cc0f33a0f8782a93bccba2a38d172c691deaeea2bdf6ca99eca"
Sep 16 12:13:54.585048 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:54.585024    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9"
Sep 16 12:13:54.687634 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:54.687601    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r65th\" (UniqueName: \"kubernetes.io/projected/5bcd29ef-5a04-4005-84a1-a9fbc8d4b391-kube-api-access-r65th\") pod \"5bcd29ef-5a04-4005-84a1-a9fbc8d4b391\" (UID: \"5bcd29ef-5a04-4005-84a1-a9fbc8d4b391\") "
Sep 16 12:13:54.687847 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:54.687676    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5bcd29ef-5a04-4005-84a1-a9fbc8d4b391-util\") pod \"5bcd29ef-5a04-4005-84a1-a9fbc8d4b391\" (UID: \"5bcd29ef-5a04-4005-84a1-a9fbc8d4b391\") "
Sep 16 12:13:54.687847 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:54.687706    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5bcd29ef-5a04-4005-84a1-a9fbc8d4b391-bundle\") pod \"5bcd29ef-5a04-4005-84a1-a9fbc8d4b391\" (UID: \"5bcd29ef-5a04-4005-84a1-a9fbc8d4b391\") "
Sep 16 12:13:54.688179 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:54.688153    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bcd29ef-5a04-4005-84a1-a9fbc8d4b391-bundle" (OuterVolumeSpecName: "bundle") pod "5bcd29ef-5a04-4005-84a1-a9fbc8d4b391" (UID: "5bcd29ef-5a04-4005-84a1-a9fbc8d4b391"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:13:54.689767 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:54.689742    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bcd29ef-5a04-4005-84a1-a9fbc8d4b391-kube-api-access-r65th" (OuterVolumeSpecName: "kube-api-access-r65th") pod "5bcd29ef-5a04-4005-84a1-a9fbc8d4b391" (UID: "5bcd29ef-5a04-4005-84a1-a9fbc8d4b391"). InnerVolumeSpecName "kube-api-access-r65th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:13:54.693052 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:54.693031    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bcd29ef-5a04-4005-84a1-a9fbc8d4b391-util" (OuterVolumeSpecName: "util") pod "5bcd29ef-5a04-4005-84a1-a9fbc8d4b391" (UID: "5bcd29ef-5a04-4005-84a1-a9fbc8d4b391"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:13:54.788312 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:54.788229    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5bcd29ef-5a04-4005-84a1-a9fbc8d4b391-util\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:13:54.788312 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:54.788268    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5bcd29ef-5a04-4005-84a1-a9fbc8d4b391-bundle\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:13:54.788312 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:54.788277    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r65th\" (UniqueName: \"kubernetes.io/projected/5bcd29ef-5a04-4005-84a1-a9fbc8d4b391-kube-api-access-r65th\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:13:55.461221 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:55.461192    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9"
Sep 16 12:13:55.461586 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:55.461194    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6ngnn9" event={"ID":"5bcd29ef-5a04-4005-84a1-a9fbc8d4b391","Type":"ContainerDied","Data":"62675df1d3ce916305b0646d6987272aae58e9d81b010f84733ca7d0412c4863"}
Sep 16 12:13:55.461586 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:13:55.461313    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62675df1d3ce916305b0646d6987272aae58e9d81b010f84733ca7d0412c4863"
Sep 16 12:14:39.009795 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.009758    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-84gmp"]
Sep 16 12:14:39.010255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010115    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7635b03-5ad7-46f5-b584-7c2439e4b196" containerName="pull"
Sep 16 12:14:39.010255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010128    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7635b03-5ad7-46f5-b584-7c2439e4b196" containerName="pull"
Sep 16 12:14:39.010255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010144    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bcd29ef-5a04-4005-84a1-a9fbc8d4b391" containerName="util"
Sep 16 12:14:39.010255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010151    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bcd29ef-5a04-4005-84a1-a9fbc8d4b391" containerName="util"
Sep 16 12:14:39.010255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010156    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4" containerName="extract"
Sep 16 12:14:39.010255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010162    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4" containerName="extract"
Sep 16 12:14:39.010255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010176    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7635b03-5ad7-46f5-b584-7c2439e4b196" containerName="extract"
Sep 16 12:14:39.010255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010181    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7635b03-5ad7-46f5-b584-7c2439e4b196" containerName="extract"
Sep 16 12:14:39.010255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010189    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7635b03-5ad7-46f5-b584-7c2439e4b196" containerName="util"
Sep 16 12:14:39.010255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010194    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7635b03-5ad7-46f5-b584-7c2439e4b196" containerName="util"
Sep 16 12:14:39.010255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010202    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c47389-88fd-4804-bf48-2bb391470716" containerName="util"
Sep 16 12:14:39.010255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010207    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c47389-88fd-4804-bf48-2bb391470716" containerName="util"
Sep 16 12:14:39.010255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010217    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4" containerName="util"
Sep 16 12:14:39.010255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010224    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4" containerName="util"
Sep 16 12:14:39.010255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010231    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c47389-88fd-4804-bf48-2bb391470716" containerName="extract"
Sep 16 12:14:39.010255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010236    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c47389-88fd-4804-bf48-2bb391470716" containerName="extract"
Sep 16 12:14:39.010255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010241    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bcd29ef-5a04-4005-84a1-a9fbc8d4b391" containerName="pull"
Sep 16 12:14:39.010255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010246    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bcd29ef-5a04-4005-84a1-a9fbc8d4b391" containerName="pull"
Sep 16 12:14:39.010255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010252    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c47389-88fd-4804-bf48-2bb391470716" containerName="pull"
Sep 16 12:14:39.010255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010257    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c47389-88fd-4804-bf48-2bb391470716" containerName="pull"
Sep 16 12:14:39.010255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010265    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bcd29ef-5a04-4005-84a1-a9fbc8d4b391" containerName="extract"
Sep 16 12:14:39.010255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010269    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bcd29ef-5a04-4005-84a1-a9fbc8d4b391" containerName="extract"
Sep 16 12:14:39.010910 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010275    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4" containerName="pull"
Sep 16 12:14:39.010910 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010280    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4" containerName="pull"
Sep 16 12:14:39.010910 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010344    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7635b03-5ad7-46f5-b584-7c2439e4b196" containerName="extract"
Sep 16 12:14:39.010910 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010352    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bcd29ef-5a04-4005-84a1-a9fbc8d4b391" containerName="extract"
Sep 16 12:14:39.010910 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010359    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c47389-88fd-4804-bf48-2bb391470716" containerName="extract"
Sep 16 12:14:39.010910 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.010363    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5dfa84f-bd71-4686-87c3-7d0b8dcbcde4" containerName="extract"
Sep 16 12:14:39.014908 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.014881    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84gmp"
Sep 16 12:14:39.018521 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.018493    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 12:14:39.018733 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.018712    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 12:14:39.018930 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.018720    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-g5zbz\""
Sep 16 12:14:39.019104 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.019082    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 16 12:14:39.019239 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.018755    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 16 12:14:39.021861 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.021839    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-84gmp"]
Sep 16 12:14:39.171262 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.171226    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/18161926-6fd0-4968-a37a-e34fe2d290ed-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-84gmp\" (UID: \"18161926-6fd0-4968-a37a-e34fe2d290ed\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84gmp"
Sep 16 12:14:39.171428 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.171291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clvwx\" (UniqueName: \"kubernetes.io/projected/18161926-6fd0-4968-a37a-e34fe2d290ed-kube-api-access-clvwx\") pod \"kuadrant-console-plugin-fbdd4f445-84gmp\" (UID: \"18161926-6fd0-4968-a37a-e34fe2d290ed\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84gmp"
Sep 16 12:14:39.171428 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.171335    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/18161926-6fd0-4968-a37a-e34fe2d290ed-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-84gmp\" (UID: \"18161926-6fd0-4968-a37a-e34fe2d290ed\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84gmp"
Sep 16 12:14:39.272827 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.272742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clvwx\" (UniqueName: \"kubernetes.io/projected/18161926-6fd0-4968-a37a-e34fe2d290ed-kube-api-access-clvwx\") pod \"kuadrant-console-plugin-fbdd4f445-84gmp\" (UID: \"18161926-6fd0-4968-a37a-e34fe2d290ed\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84gmp"
Sep 16 12:14:39.272827 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.272794    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/18161926-6fd0-4968-a37a-e34fe2d290ed-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-84gmp\" (UID: \"18161926-6fd0-4968-a37a-e34fe2d290ed\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84gmp"
Sep 16 12:14:39.273009 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.272846    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/18161926-6fd0-4968-a37a-e34fe2d290ed-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-84gmp\" (UID: \"18161926-6fd0-4968-a37a-e34fe2d290ed\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84gmp"
Sep 16 12:14:39.273402 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.273366    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/18161926-6fd0-4968-a37a-e34fe2d290ed-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-84gmp\" (UID: \"18161926-6fd0-4968-a37a-e34fe2d290ed\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84gmp"
Sep 16 12:14:39.275244 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.275225    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/18161926-6fd0-4968-a37a-e34fe2d290ed-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-84gmp\" (UID: \"18161926-6fd0-4968-a37a-e34fe2d290ed\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84gmp"
Sep 16 12:14:39.283417 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.283385    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clvwx\" (UniqueName: \"kubernetes.io/projected/18161926-6fd0-4968-a37a-e34fe2d290ed-kube-api-access-clvwx\") pod \"kuadrant-console-plugin-fbdd4f445-84gmp\" (UID: \"18161926-6fd0-4968-a37a-e34fe2d290ed\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84gmp"
Sep 16 12:14:39.325628 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.325600    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84gmp"
Sep 16 12:14:39.597608 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.597581    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-84gmp"]
Sep 16 12:14:39.598989 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:14:39.598958    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18161926_6fd0_4968_a37a_e34fe2d290ed.slice/crio-265b0a1c52f053032ea70d6bd908f56393e05201bdcf99f790af605c00bb5412 WatchSource:0}: Error finding container 265b0a1c52f053032ea70d6bd908f56393e05201bdcf99f790af605c00bb5412: Status 404 returned error can't find the container with id 265b0a1c52f053032ea70d6bd908f56393e05201bdcf99f790af605c00bb5412
Sep 16 12:14:39.686492 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:39.686455    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84gmp" event={"ID":"18161926-6fd0-4968-a37a-e34fe2d290ed","Type":"ContainerStarted","Data":"265b0a1c52f053032ea70d6bd908f56393e05201bdcf99f790af605c00bb5412"}
Sep 16 12:14:44.720182 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:44.720142    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84gmp" event={"ID":"18161926-6fd0-4968-a37a-e34fe2d290ed","Type":"ContainerStarted","Data":"2f8c7098bc55fe4229ec6a563beffc597a3870ed907bd100eab3bfdec4b4f10a"}
Sep 16 12:14:44.737589 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:14:44.737537    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84gmp" podStartSLOduration=2.271996515 podStartE2EDuration="6.737520847s" totalImagesPullingTime="4.465524332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:14:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:14:44.73661344 +0000 UTC m=+678.437684777" watchObservedRunningTime="2026-09-16 12:14:44.737520847 +0000 UTC m=+678.438592172"
Sep 16 12:15:21.662862 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.662787    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"]
Sep 16 12:15:21.681920 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.681894    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"]
Sep 16 12:15:21.682066 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.681985    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.684474 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.684453    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zfcwg\""
Sep 16 12:15:21.844081 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.844050    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab958a3-431d-4406-953e-c5440bb50bb5-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.844268 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.844089    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab958a3-431d-4406-953e-c5440bb50bb5-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.844268 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.844107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab958a3-431d-4406-953e-c5440bb50bb5-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.844268 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.844174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab958a3-431d-4406-953e-c5440bb50bb5-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.844268 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.844227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg2x8\" (UniqueName: \"kubernetes.io/projected/6ab958a3-431d-4406-953e-c5440bb50bb5-kube-api-access-sg2x8\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.844268 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.844255    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab958a3-431d-4406-953e-c5440bb50bb5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.844471 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.844275    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab958a3-431d-4406-953e-c5440bb50bb5-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.844471 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.844289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab958a3-431d-4406-953e-c5440bb50bb5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.844471 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.844309    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab958a3-431d-4406-953e-c5440bb50bb5-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.945419 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.945318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab958a3-431d-4406-953e-c5440bb50bb5-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.945419 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.945393    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab958a3-431d-4406-953e-c5440bb50bb5-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.945419 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.945425    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab958a3-431d-4406-953e-c5440bb50bb5-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.945742 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.945439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab958a3-431d-4406-953e-c5440bb50bb5-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.945742 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.945477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab958a3-431d-4406-953e-c5440bb50bb5-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.945742 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.945509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg2x8\" (UniqueName: \"kubernetes.io/projected/6ab958a3-431d-4406-953e-c5440bb50bb5-kube-api-access-sg2x8\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.945742 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.945542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab958a3-431d-4406-953e-c5440bb50bb5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.945742 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.945574    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab958a3-431d-4406-953e-c5440bb50bb5-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.945742 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.945604    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab958a3-431d-4406-953e-c5440bb50bb5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.945981 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.945801    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab958a3-431d-4406-953e-c5440bb50bb5-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.945981 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.945920    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ab958a3-431d-4406-953e-c5440bb50bb5-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.946129 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.946111    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ab958a3-431d-4406-953e-c5440bb50bb5-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.946197 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.946153    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ab958a3-431d-4406-953e-c5440bb50bb5-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.946253 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.946234    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ab958a3-431d-4406-953e-c5440bb50bb5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.947756 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.947740    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ab958a3-431d-4406-953e-c5440bb50bb5-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.947985 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.947967    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ab958a3-431d-4406-953e-c5440bb50bb5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.952925 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.952902    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ab958a3-431d-4406-953e-c5440bb50bb5-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.953094 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.953076    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg2x8\" (UniqueName: \"kubernetes.io/projected/6ab958a3-431d-4406-953e-c5440bb50bb5-kube-api-access-sg2x8\") pod \"maas-default-gateway-openshift-default-78bb78ff78-h6qjt\" (UID: \"6ab958a3-431d-4406-953e-c5440bb50bb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:21.992095 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:21.992068    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:22.270424 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:22.270396    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"]
Sep 16 12:15:22.272569 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:15:22.272540    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab958a3_431d_4406_953e_c5440bb50bb5.slice/crio-1745dc143492a1aa1374f80f3b98f3cd5479117acc56d26ba697f16e2633b425 WatchSource:0}: Error finding container 1745dc143492a1aa1374f80f3b98f3cd5479117acc56d26ba697f16e2633b425: Status 404 returned error can't find the container with id 1745dc143492a1aa1374f80f3b98f3cd5479117acc56d26ba697f16e2633b425
Sep 16 12:15:22.274756 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:22.274711    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:15:22.274856 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:22.274808    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:15:22.274898 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:22.274853    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:15:22.915368 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:22.915330    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt" event={"ID":"6ab958a3-431d-4406-953e-c5440bb50bb5","Type":"ContainerStarted","Data":"65e8617f05738bd697ad14a9a5ddf16509ab39f8c6bdcc2134b9ba9c823e5a77"}
Sep 16 12:15:22.915864 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:22.915379    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt" event={"ID":"6ab958a3-431d-4406-953e-c5440bb50bb5","Type":"ContainerStarted","Data":"1745dc143492a1aa1374f80f3b98f3cd5479117acc56d26ba697f16e2633b425"}
Sep 16 12:15:22.937073 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:22.937025    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt" podStartSLOduration=1.9370098599999999 podStartE2EDuration="1.93700986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:15:22.934190789 +0000 UTC m=+716.635262126" watchObservedRunningTime="2026-09-16 12:15:22.93700986 +0000 UTC m=+716.638081230"
Sep 16 12:15:22.992286 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:22.992258    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:23.997262 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:23.997231    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:24.925300 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:24.925260    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:24.926314 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:24.926291    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:15:35.684565 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:35.684518    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:15:35.688164 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:35.688146    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-flrsc"
Sep 16 12:15:35.690743 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:35.690721    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 12:15:35.696828 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:35.696803    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:15:35.724870 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:35.724834    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:15:35.762100 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:35.762070    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlt4j\" (UniqueName: \"kubernetes.io/projected/e5be99d1-3eed-4b4f-96d4-813e2b8a6775-kube-api-access-nlt4j\") pod \"limitador-limitador-84994f7ffc-flrsc\" (UID: \"e5be99d1-3eed-4b4f-96d4-813e2b8a6775\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-flrsc"
Sep 16 12:15:35.762233 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:35.762107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e5be99d1-3eed-4b4f-96d4-813e2b8a6775-config-file\") pod \"limitador-limitador-84994f7ffc-flrsc\" (UID: \"e5be99d1-3eed-4b4f-96d4-813e2b8a6775\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-flrsc"
Sep 16 12:15:35.862673 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:35.862634    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlt4j\" (UniqueName: \"kubernetes.io/projected/e5be99d1-3eed-4b4f-96d4-813e2b8a6775-kube-api-access-nlt4j\") pod \"limitador-limitador-84994f7ffc-flrsc\" (UID: \"e5be99d1-3eed-4b4f-96d4-813e2b8a6775\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-flrsc"
Sep 16 12:15:35.862824 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:35.862701    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e5be99d1-3eed-4b4f-96d4-813e2b8a6775-config-file\") pod \"limitador-limitador-84994f7ffc-flrsc\" (UID: \"e5be99d1-3eed-4b4f-96d4-813e2b8a6775\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-flrsc"
Sep 16 12:15:35.863314 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:35.863294    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e5be99d1-3eed-4b4f-96d4-813e2b8a6775-config-file\") pod \"limitador-limitador-84994f7ffc-flrsc\" (UID: \"e5be99d1-3eed-4b4f-96d4-813e2b8a6775\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-flrsc"
Sep 16 12:15:35.870159 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:35.870136    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlt4j\" (UniqueName: \"kubernetes.io/projected/e5be99d1-3eed-4b4f-96d4-813e2b8a6775-kube-api-access-nlt4j\") pod \"limitador-limitador-84994f7ffc-flrsc\" (UID: \"e5be99d1-3eed-4b4f-96d4-813e2b8a6775\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-flrsc"
Sep 16 12:15:35.999131 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:35.999040    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-flrsc"
Sep 16 12:15:36.270222 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:36.270196    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:15:36.271886 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:15:36.271859    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5be99d1_3eed_4b4f_96d4_813e2b8a6775.slice/crio-16466df5e4385fd374de9949b080ae1dc603f45ec2c9e5d0c59335eda20a1034 WatchSource:0}: Error finding container 16466df5e4385fd374de9949b080ae1dc603f45ec2c9e5d0c59335eda20a1034: Status 404 returned error can't find the container with id 16466df5e4385fd374de9949b080ae1dc603f45ec2c9e5d0c59335eda20a1034
Sep 16 12:15:36.995314 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:36.995273    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-flrsc" event={"ID":"e5be99d1-3eed-4b4f-96d4-813e2b8a6775","Type":"ContainerStarted","Data":"16466df5e4385fd374de9949b080ae1dc603f45ec2c9e5d0c59335eda20a1034"}
Sep 16 12:15:39.006913 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:39.006873    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-flrsc" event={"ID":"e5be99d1-3eed-4b4f-96d4-813e2b8a6775","Type":"ContainerStarted","Data":"087940f43b9f144523134a0215841f4d10772509a9eef969924d4c3f2a85c9b4"}
Sep 16 12:15:39.007286 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:39.007010    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-flrsc"
Sep 16 12:15:39.022454 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:39.022407    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-flrsc" podStartSLOduration=2.127073572 podStartE2EDuration="4.022389903s" totalImagesPullingTime="1.895316331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:15:39.021294571 +0000 UTC m=+732.722365911" watchObservedRunningTime="2026-09-16 12:15:39.022389903 +0000 UTC m=+732.723461231"
Sep 16 12:15:50.013584 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:15:50.013545    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-flrsc"
Sep 16 12:16:52.335337 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:16:52.335250    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:17:01.452938 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:01.452819    2579 generic.go:356] "Generic (PLEG): container finished" podID="6ab958a3-431d-4406-953e-c5440bb50bb5" containerID="65e8617f05738bd697ad14a9a5ddf16509ab39f8c6bdcc2134b9ba9c823e5a77" exitCode=0
Sep 16 12:17:01.452938 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:01.452898    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt" event={"ID":"6ab958a3-431d-4406-953e-c5440bb50bb5","Type":"ContainerDied","Data":"65e8617f05738bd697ad14a9a5ddf16509ab39f8c6bdcc2134b9ba9c823e5a77"}
Sep 16 12:17:01.453570 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:01.453390    2579 scope.go:117] "RemoveContainer" containerID="65e8617f05738bd697ad14a9a5ddf16509ab39f8c6bdcc2134b9ba9c823e5a77"
Sep 16 12:17:01.454513 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:01.454480    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:17:01.454591 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:01.454574    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:17:01.454638 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:01.454624    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:17:02.460185 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:02.460148    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt" event={"ID":"6ab958a3-431d-4406-953e-c5440bb50bb5","Type":"ContainerStarted","Data":"5c145b277ed9fdcf11c0866026476f8a513f7f534a64be3cba54eb71db127bfa"}
Sep 16 12:17:02.992802 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:02.992766    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:17:02.992993 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:02.992826    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-h6qjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 16 12:17:02.992993 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:02.992911    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt" podUID="6ab958a3-431d-4406-953e-c5440bb50bb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 16 12:17:03.993397 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:03.993358    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-h6qjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 16 12:17:03.993986 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:03.993952    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt" podUID="6ab958a3-431d-4406-953e-c5440bb50bb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 16 12:17:04.993010 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:04.992973    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-h6qjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 16 12:17:04.993290 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:04.993261    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt" podUID="6ab958a3-431d-4406-953e-c5440bb50bb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 16 12:17:05.993492 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:05.993452    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-h6qjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 16 12:17:05.994105 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:05.994071    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt" podUID="6ab958a3-431d-4406-953e-c5440bb50bb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 16 12:17:06.993158 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:06.993104    2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-h6qjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 16 12:17:06.993368 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:06.993184    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt" podUID="6ab958a3-431d-4406-953e-c5440bb50bb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 16 12:17:07.996215 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:07.996181    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:17:07.996606 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:07.996463    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:17:07.997133 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:07.997103    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-h6qjt"
Sep 16 12:17:11.250901 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:11.250864    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:17:35.961907 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:35.961875    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:17:47.444815 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.444778    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"]
Sep 16 12:17:47.448413 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.448396    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"
Sep 16 12:17:47.451297 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.451276    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 12:17:47.452697 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.452682    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 12:17:47.452803 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.452695    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 16 12:17:47.452803 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.452698    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-c49mq\""
Sep 16 12:17:47.458099 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.458075    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"]
Sep 16 12:17:47.506812 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.506767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25e0b03f-c77f-4a62-9eff-0a4c11331d8f-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx\" (UID: \"25e0b03f-c77f-4a62-9eff-0a4c11331d8f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"
Sep 16 12:17:47.506982 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.506832    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25e0b03f-c77f-4a62-9eff-0a4c11331d8f-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx\" (UID: \"25e0b03f-c77f-4a62-9eff-0a4c11331d8f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"
Sep 16 12:17:47.506982 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.506899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25e0b03f-c77f-4a62-9eff-0a4c11331d8f-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx\" (UID: \"25e0b03f-c77f-4a62-9eff-0a4c11331d8f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"
Sep 16 12:17:47.506982 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.506963    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h6vt\" (UniqueName: \"kubernetes.io/projected/25e0b03f-c77f-4a62-9eff-0a4c11331d8f-kube-api-access-4h6vt\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx\" (UID: \"25e0b03f-c77f-4a62-9eff-0a4c11331d8f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"
Sep 16 12:17:47.507151 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.507042    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25e0b03f-c77f-4a62-9eff-0a4c11331d8f-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx\" (UID: \"25e0b03f-c77f-4a62-9eff-0a4c11331d8f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"
Sep 16 12:17:47.507151 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.507094    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25e0b03f-c77f-4a62-9eff-0a4c11331d8f-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx\" (UID: \"25e0b03f-c77f-4a62-9eff-0a4c11331d8f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"
Sep 16 12:17:47.608193 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.608152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25e0b03f-c77f-4a62-9eff-0a4c11331d8f-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx\" (UID: \"25e0b03f-c77f-4a62-9eff-0a4c11331d8f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"
Sep 16 12:17:47.608376 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.608204    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25e0b03f-c77f-4a62-9eff-0a4c11331d8f-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx\" (UID: \"25e0b03f-c77f-4a62-9eff-0a4c11331d8f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"
Sep 16 12:17:47.608376 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.608239    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25e0b03f-c77f-4a62-9eff-0a4c11331d8f-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx\" (UID: \"25e0b03f-c77f-4a62-9eff-0a4c11331d8f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"
Sep 16 12:17:47.608376 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.608264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h6vt\" (UniqueName: \"kubernetes.io/projected/25e0b03f-c77f-4a62-9eff-0a4c11331d8f-kube-api-access-4h6vt\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx\" (UID: \"25e0b03f-c77f-4a62-9eff-0a4c11331d8f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"
Sep 16 12:17:47.608376 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.608317    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25e0b03f-c77f-4a62-9eff-0a4c11331d8f-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx\" (UID: \"25e0b03f-c77f-4a62-9eff-0a4c11331d8f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"
Sep 16 12:17:47.608376 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.608357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25e0b03f-c77f-4a62-9eff-0a4c11331d8f-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx\" (UID: \"25e0b03f-c77f-4a62-9eff-0a4c11331d8f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"
Sep 16 12:17:47.608743 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.608720    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25e0b03f-c77f-4a62-9eff-0a4c11331d8f-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx\" (UID: \"25e0b03f-c77f-4a62-9eff-0a4c11331d8f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"
Sep 16 12:17:47.608823 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.608772    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25e0b03f-c77f-4a62-9eff-0a4c11331d8f-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx\" (UID: \"25e0b03f-c77f-4a62-9eff-0a4c11331d8f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"
Sep 16 12:17:47.608877 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.608821    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25e0b03f-c77f-4a62-9eff-0a4c11331d8f-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx\" (UID: \"25e0b03f-c77f-4a62-9eff-0a4c11331d8f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"
Sep 16 12:17:47.610395 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.610379    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25e0b03f-c77f-4a62-9eff-0a4c11331d8f-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx\" (UID: \"25e0b03f-c77f-4a62-9eff-0a4c11331d8f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"
Sep 16 12:17:47.610737 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.610721    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25e0b03f-c77f-4a62-9eff-0a4c11331d8f-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx\" (UID: \"25e0b03f-c77f-4a62-9eff-0a4c11331d8f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"
Sep 16 12:17:47.618198 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.618176    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h6vt\" (UniqueName: \"kubernetes.io/projected/25e0b03f-c77f-4a62-9eff-0a4c11331d8f-kube-api-access-4h6vt\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx\" (UID: \"25e0b03f-c77f-4a62-9eff-0a4c11331d8f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"
Sep 16 12:17:47.759535 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:47.759423    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"
Sep 16 12:17:48.034096 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:48.034072    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"]
Sep 16 12:17:48.035832 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:17:48.035801    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25e0b03f_c77f_4a62_9eff_0a4c11331d8f.slice/crio-8877718f400b2bc3a0e1c96ea0e16c35e1d5010893228df2783831849ffa28ea WatchSource:0}: Error finding container 8877718f400b2bc3a0e1c96ea0e16c35e1d5010893228df2783831849ffa28ea: Status 404 returned error can't find the container with id 8877718f400b2bc3a0e1c96ea0e16c35e1d5010893228df2783831849ffa28ea
Sep 16 12:17:48.459711 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:48.459674    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:17:48.722689 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:48.722570    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx" event={"ID":"25e0b03f-c77f-4a62-9eff-0a4c11331d8f","Type":"ContainerStarted","Data":"8877718f400b2bc3a0e1c96ea0e16c35e1d5010893228df2783831849ffa28ea"}
Sep 16 12:17:49.729284 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:49.729247    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx" event={"ID":"25e0b03f-c77f-4a62-9eff-0a4c11331d8f","Type":"ContainerStarted","Data":"aa99dad2bf7dcbf7ed4abda4111adfc6f06d8a12b1d6116bc446f96d61800409"}
Sep 16 12:17:49.746967 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:49.746917    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx" podStartSLOduration=1.732195023 podStartE2EDuration="2.746902872s" totalImagesPullingTime="1.014707849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:17:49.745095573 +0000 UTC m=+863.446166902" watchObservedRunningTime="2026-09-16 12:17:49.746902872 +0000 UTC m=+863.447974198"
Sep 16 12:17:57.760185 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:57.760147    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"
Sep 16 12:17:57.760582 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:57.760205    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"
Sep 16 12:17:57.772644 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:57.772620    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"
Sep 16 12:17:57.787465 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:57.787442    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx"
Sep 16 12:17:59.959970 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:17:59.959929    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:18:10.456757 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:18:10.456723    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:18:19.059485 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:18:19.059410    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:18:26.171435 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:18:26.171400    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:20:00.760866 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.760831    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"]
Sep 16 12:20:00.764114 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.764099    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.767402 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.767384    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f79e22-openshift-default-dockercfg-mrvtk\""
Sep 16 12:20:00.782964 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.782934    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"]
Sep 16 12:20:00.786798 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.786764    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istiod-ca-cert\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.786925 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.786837    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istio-data\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.786925 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.786876    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istio-token\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.786925 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.786903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istio-envoy\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.787109 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.786982    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istio-podinfo\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.787109 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.787048    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlqwv\" (UniqueName: \"kubernetes.io/projected/f35bc2fb-f32f-4f0f-8e78-918f927616e3-kube-api-access-rlqwv\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.787109 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.787088    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-workload-certs\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.787209 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.787126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-workload-socket\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.787209 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.787147    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-credential-socket\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.889393 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.889347    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istiod-ca-cert\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.889585 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.889414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istio-data\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.889585 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.889452    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istio-token\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.889585 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.889485    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istio-envoy\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.889585 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.889538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istio-podinfo\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.889885 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.889599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlqwv\" (UniqueName: \"kubernetes.io/projected/f35bc2fb-f32f-4f0f-8e78-918f927616e3-kube-api-access-rlqwv\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.889885 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.889637    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-workload-certs\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.889885 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.889720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-workload-socket\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.889885 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.889756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-credential-socket\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.890264 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.890235    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-credential-socket\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.890964 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.890935    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istiod-ca-cert\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.891269 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.891245    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istio-data\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.896676 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.894515    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istio-envoy\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.898455 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.896788    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-workload-certs\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.898455 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.896844    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-workload-socket\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.898455 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.897611    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istio-podinfo\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.903632 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.903594    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istio-token\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.908595 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.908567    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"]
Sep 16 12:20:00.913321 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.913298    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:00.916098 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.916077    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-739ee7-openshift-default-dockercfg-7fp98\""
Sep 16 12:20:00.921267 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.921235    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlqwv\" (UniqueName: \"kubernetes.io/projected/f35bc2fb-f32f-4f0f-8e78-918f927616e3-kube-api-access-rlqwv\") pod \"e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") " pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:00.922696 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.922640    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"]
Sep 16 12:20:00.990956 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.990920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ade63ee-3184-46df-ada2-55a290bcbefc-istio-token\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:00.991121 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.990971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-workload-socket\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:00.991121 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.991026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szgdf\" (UniqueName: \"kubernetes.io/projected/5ade63ee-3184-46df-ada2-55a290bcbefc-kube-api-access-szgdf\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:00.991121 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.991051    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-istio-envoy\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:00.991121 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.991077    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ade63ee-3184-46df-ada2-55a290bcbefc-istio-podinfo\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:00.991259 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.991127    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-istio-data\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:00.991259 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.991165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-credential-socket\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:00.991259 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.991210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ade63ee-3184-46df-ada2-55a290bcbefc-istiod-ca-cert\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:00.991259 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:00.991243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-workload-certs\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:01.074700 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.074567    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:01.091836 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.091805    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ade63ee-3184-46df-ada2-55a290bcbefc-istiod-ca-cert\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:01.092015 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.091848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-workload-certs\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:01.092015 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.091924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ade63ee-3184-46df-ada2-55a290bcbefc-istio-token\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:01.092015 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.091961    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-workload-socket\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:01.092015 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.091993    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szgdf\" (UniqueName: \"kubernetes.io/projected/5ade63ee-3184-46df-ada2-55a290bcbefc-kube-api-access-szgdf\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:01.092217 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.092034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-istio-envoy\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:01.092217 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.092062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ade63ee-3184-46df-ada2-55a290bcbefc-istio-podinfo\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:01.092217 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.092089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-istio-data\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:01.092217 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.092115    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-credential-socket\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:01.092400 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.092355    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-workload-certs\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:01.092688 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.092636    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-workload-socket\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:01.092931 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.092903    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ade63ee-3184-46df-ada2-55a290bcbefc-istiod-ca-cert\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:01.093012 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.092909    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-credential-socket\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:01.093012 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.092906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-istio-data\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:01.094635 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.094613    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-istio-envoy\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:01.095440 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.095419    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ade63ee-3184-46df-ada2-55a290bcbefc-istio-podinfo\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:01.101174 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.101145    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ade63ee-3184-46df-ada2-55a290bcbefc-istio-token\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:01.101297 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.101278    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szgdf\" (UniqueName: \"kubernetes.io/projected/5ade63ee-3184-46df-ada2-55a290bcbefc-kube-api-access-szgdf\") pod \"e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") " pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:01.236745 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.236704    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:01.358334 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.358299    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"]
Sep 16 12:20:01.359422 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:20:01.359396    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35bc2fb_f32f_4f0f_8e78_918f927616e3.slice/crio-0005541f307559c3ca7149bb87bd1cd25d5c63761be9f4d94ac4992206031c5e WatchSource:0}: Error finding container 0005541f307559c3ca7149bb87bd1cd25d5c63761be9f4d94ac4992206031c5e: Status 404 returned error can't find the container with id 0005541f307559c3ca7149bb87bd1cd25d5c63761be9f4d94ac4992206031c5e
Sep 16 12:20:01.361737 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.361703    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:01.361827 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.361771    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:01.361827 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.361798    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:01.458347 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.458306    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx" event={"ID":"f35bc2fb-f32f-4f0f-8e78-918f927616e3","Type":"ContainerStarted","Data":"255281c2fb0468af70a0473be5d68852a8712a74f640af6ef45ea760c676c037"}
Sep 16 12:20:01.458499 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.458363    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx" event={"ID":"f35bc2fb-f32f-4f0f-8e78-918f927616e3","Type":"ContainerStarted","Data":"0005541f307559c3ca7149bb87bd1cd25d5c63761be9f4d94ac4992206031c5e"}
Sep 16 12:20:01.479350 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.479287    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx" podStartSLOduration=1.4792662220000001 podStartE2EDuration="1.479266222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:01.477954622 +0000 UTC m=+995.179025961" watchObservedRunningTime="2026-09-16 12:20:01.479266222 +0000 UTC m=+995.180337546"
Sep 16 12:20:01.515969 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.515939    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"]
Sep 16 12:20:01.518607 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:20:01.518578    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ade63ee_3184_46df_ada2_55a290bcbefc.slice/crio-d044b52a33f2fb4ce4ba3c3aad9389a1da5df265f7335a3a369c5b9de8e622b1 WatchSource:0}: Error finding container d044b52a33f2fb4ce4ba3c3aad9389a1da5df265f7335a3a369c5b9de8e622b1: Status 404 returned error can't find the container with id d044b52a33f2fb4ce4ba3c3aad9389a1da5df265f7335a3a369c5b9de8e622b1
Sep 16 12:20:01.520956 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.520924    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:01.521029 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.520990    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:01.521029 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:01.521020    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:02.074723 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:02.074686    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:02.079903 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:02.079874    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:02.466074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:02.466037    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg" event={"ID":"5ade63ee-3184-46df-ada2-55a290bcbefc","Type":"ContainerStarted","Data":"da5db95d12256e348e633bc22c6c62e9ece315dd0cb228c175d796fe6ac3f0b1"}
Sep 16 12:20:02.466226 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:02.466095    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg" event={"ID":"5ade63ee-3184-46df-ada2-55a290bcbefc","Type":"ContainerStarted","Data":"d044b52a33f2fb4ce4ba3c3aad9389a1da5df265f7335a3a369c5b9de8e622b1"}
Sep 16 12:20:02.466300 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:02.466283    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:02.467406 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:02.467384    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:02.489356 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:02.489297    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg" podStartSLOduration=2.489281025 podStartE2EDuration="2.489281025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:02.488135154 +0000 UTC m=+996.189206480" watchObservedRunningTime="2026-09-16 12:20:02.489281025 +0000 UTC m=+996.190352350"
Sep 16 12:20:02.513850 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:02.513813    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7892-73db19"]
Sep 16 12:20:02.517614 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:02.517586    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7892-73db19"
Sep 16 12:20:02.520368 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:02.520343    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6btc9\""
Sep 16 12:20:02.526793 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:02.526694    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7892-73db19"]
Sep 16 12:20:02.604625 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:02.604581    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjwww\" (UniqueName: \"kubernetes.io/projected/9e0d3e92-11dc-4bdf-a34f-5b2e313f45f9-kube-api-access-zjwww\") pod \"test-curl-7892-73db19\" (UID: \"9e0d3e92-11dc-4bdf-a34f-5b2e313f45f9\") " pod="opendatahub/test-curl-7892-73db19"
Sep 16 12:20:02.705587 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:02.705540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjwww\" (UniqueName: \"kubernetes.io/projected/9e0d3e92-11dc-4bdf-a34f-5b2e313f45f9-kube-api-access-zjwww\") pod \"test-curl-7892-73db19\" (UID: \"9e0d3e92-11dc-4bdf-a34f-5b2e313f45f9\") " pod="opendatahub/test-curl-7892-73db19"
Sep 16 12:20:02.720680 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:02.720607    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjwww\" (UniqueName: \"kubernetes.io/projected/9e0d3e92-11dc-4bdf-a34f-5b2e313f45f9-kube-api-access-zjwww\") pod \"test-curl-7892-73db19\" (UID: \"9e0d3e92-11dc-4bdf-a34f-5b2e313f45f9\") " pod="opendatahub/test-curl-7892-73db19"
Sep 16 12:20:02.831030 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:02.830991    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7892-73db19"
Sep 16 12:20:03.103938 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:03.103908    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7892-73db19"]
Sep 16 12:20:03.105307 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:20:03.105264    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e0d3e92_11dc_4bdf_a34f_5b2e313f45f9.slice/crio-d631f0f594afaae1ea66104632159a3c408253969c627a38bb9aebbb63275ba0 WatchSource:0}: Error finding container d631f0f594afaae1ea66104632159a3c408253969c627a38bb9aebbb63275ba0: Status 404 returned error can't find the container with id d631f0f594afaae1ea66104632159a3c408253969c627a38bb9aebbb63275ba0
Sep 16 12:20:03.107394 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:03.107375    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:20:03.173513 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:03.173478    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:20:03.237815 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:03.237776    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:03.243256 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:03.243230    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:03.473789 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:03.473686    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7892-73db19" event={"ID":"9e0d3e92-11dc-4bdf-a34f-5b2e313f45f9","Type":"ContainerStarted","Data":"d631f0f594afaae1ea66104632159a3c408253969c627a38bb9aebbb63275ba0"}
Sep 16 12:20:03.474153 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:03.474111    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:03.474991 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:03.474968    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:05.488840 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:05.488749    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7892-73db19" event={"ID":"9e0d3e92-11dc-4bdf-a34f-5b2e313f45f9","Type":"ContainerStarted","Data":"11261477603b413d354d9a75b0f547b60ecfbb465776de1174703ac527252180"}
Sep 16 12:20:05.503826 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:05.503779    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7892-73db19" podStartSLOduration=1.471095278 podStartE2EDuration="3.503764631s" totalImagesPullingTime="2.032669353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:05.501776762 +0000 UTC m=+999.202848087" watchObservedRunningTime="2026-09-16 12:20:05.503764631 +0000 UTC m=+999.204835956"
Sep 16 12:20:06.085107 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:06.085064    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7892-73db19"]
Sep 16 12:20:06.088882 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:06.088853    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7892-73db19"]
Sep 16 12:20:06.210445 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:06.210362    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7892-bb81f9"]
Sep 16 12:20:06.210784 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:06.210770    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e0d3e92-11dc-4bdf-a34f-5b2e313f45f9" containerName="test-curl-7892-73db19"
Sep 16 12:20:06.210784 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:06.210785    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e0d3e92-11dc-4bdf-a34f-5b2e313f45f9" containerName="test-curl-7892-73db19"
Sep 16 12:20:06.210881 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:06.210863    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e0d3e92-11dc-4bdf-a34f-5b2e313f45f9" containerName="test-curl-7892-73db19"
Sep 16 12:20:06.214318 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:06.214292    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7892-bb81f9"
Sep 16 12:20:06.217121 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:06.217094    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7892-bb81f9"]
Sep 16 12:20:06.341982 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:06.341944    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r68v6\" (UniqueName: \"kubernetes.io/projected/eb81778a-86b8-46b8-b640-92d22665ca31-kube-api-access-r68v6\") pod \"test-curl-7892-bb81f9\" (UID: \"eb81778a-86b8-46b8-b640-92d22665ca31\") " pod="opendatahub/test-curl-7892-bb81f9"
Sep 16 12:20:06.443176 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:06.443136    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r68v6\" (UniqueName: \"kubernetes.io/projected/eb81778a-86b8-46b8-b640-92d22665ca31-kube-api-access-r68v6\") pod \"test-curl-7892-bb81f9\" (UID: \"eb81778a-86b8-46b8-b640-92d22665ca31\") " pod="opendatahub/test-curl-7892-bb81f9"
Sep 16 12:20:06.450702 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:06.450652    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r68v6\" (UniqueName: \"kubernetes.io/projected/eb81778a-86b8-46b8-b640-92d22665ca31-kube-api-access-r68v6\") pod \"test-curl-7892-bb81f9\" (UID: \"eb81778a-86b8-46b8-b640-92d22665ca31\") " pod="opendatahub/test-curl-7892-bb81f9"
Sep 16 12:20:06.495749 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:20:06.495642    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7892-73db19" container="test-curl-7892-73db19"
Sep 16 12:20:06.524680 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:06.524631    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7892-bb81f9"
Sep 16 12:20:06.797006 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:06.796983    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7892-bb81f9"]
Sep 16 12:20:06.798468 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:20:06.798443    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb81778a_86b8_46b8_b640_92d22665ca31.slice/crio-b8923c2a0a7c6d36fd68ad2afd931aceac4ec4ae1b3bb630c34cffd8d4082995 WatchSource:0}: Error finding container b8923c2a0a7c6d36fd68ad2afd931aceac4ec4ae1b3bb630c34cffd8d4082995: Status 404 returned error can't find the container with id b8923c2a0a7c6d36fd68ad2afd931aceac4ec4ae1b3bb630c34cffd8d4082995
Sep 16 12:20:07.502537 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:07.502499    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7892-bb81f9" event={"ID":"eb81778a-86b8-46b8-b640-92d22665ca31","Type":"ContainerStarted","Data":"d3864bc6d38f929c3b0ffb94f917468e9aa13af01666cfa0b4857ed1b44bfe80"}
Sep 16 12:20:07.502537 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:07.502546    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7892-bb81f9" event={"ID":"eb81778a-86b8-46b8-b640-92d22665ca31","Type":"ContainerStarted","Data":"b8923c2a0a7c6d36fd68ad2afd931aceac4ec4ae1b3bb630c34cffd8d4082995"}
Sep 16 12:20:07.503058 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:07.502593    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7892-73db19" podUID="9e0d3e92-11dc-4bdf-a34f-5b2e313f45f9" containerName="test-curl-7892-73db19" containerID="cri-o://11261477603b413d354d9a75b0f547b60ecfbb465776de1174703ac527252180" gracePeriod=2
Sep 16 12:20:07.519255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:07.519205    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7892-bb81f9" podStartSLOduration=1.519189768 podStartE2EDuration="1.519189768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:07.516713486 +0000 UTC m=+1001.217784814" watchObservedRunningTime="2026-09-16 12:20:07.519189768 +0000 UTC m=+1001.220261095"
Sep 16 12:20:08.117817 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:08.117777    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7892-bb81f9"]
Sep 16 12:20:08.121439 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:08.121411    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7892-bb81f9"]
Sep 16 12:20:08.508140 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:20:08.508103    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7892-bb81f9" container="test-curl-7892-bb81f9"
Sep 16 12:20:08.938125 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:08.938089    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"]
Sep 16 12:20:08.938524 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:08.938509    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb81778a-86b8-46b8-b640-92d22665ca31" containerName="test-curl-7892-bb81f9"
Sep 16 12:20:08.938580 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:08.938529    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb81778a-86b8-46b8-b640-92d22665ca31" containerName="test-curl-7892-bb81f9"
Sep 16 12:20:08.938613 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:08.938591    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb81778a-86b8-46b8-b640-92d22665ca31" containerName="test-curl-7892-bb81f9"
Sep 16 12:20:08.941877 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:08.941855    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:08.946252 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:08.946224    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-00a141-openshift-default-dockercfg-pn8hr\""
Sep 16 12:20:08.956006 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:08.955986    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"]
Sep 16 12:20:09.065498 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.065463    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istiod-ca-cert\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.065498 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.065508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istio-podinfo\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.065743 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.065537    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istio-envoy\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.065743 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.065623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istio-data\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.065743 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.065671    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istio-token\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.065743 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.065699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-workload-certs\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.065892 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.065753    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-credential-socket\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.065892 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.065852    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g4kj\" (UniqueName: \"kubernetes.io/projected/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-kube-api-access-5g4kj\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.065961 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.065952    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-workload-socket\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.166998 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.166958    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g4kj\" (UniqueName: \"kubernetes.io/projected/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-kube-api-access-5g4kj\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.167180 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.167028    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-workload-socket\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.167180 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.167055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istiod-ca-cert\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.167180 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.167073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istio-podinfo\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.167180 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.167091    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istio-envoy\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.167180 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.167121    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istio-data\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.167180 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.167169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istio-token\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.167492 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.167188    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-workload-certs\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.167492 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.167242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-credential-socket\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.167604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.167556    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-workload-socket\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.167698 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.167645    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-workload-certs\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.167817 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.167789    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istio-data\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.167946 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.167874    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istiod-ca-cert\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.168018 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.167948    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-credential-socket\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.169485 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.169460    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istio-envoy\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.169702 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.169681    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istio-podinfo\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.175586 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.175563    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g4kj\" (UniqueName: \"kubernetes.io/projected/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-kube-api-access-5g4kj\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.175954 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.175932    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istio-token\") pod \"e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") " pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.252314 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.252237    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:09.514532 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.514488    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7892-bb81f9" podUID="eb81778a-86b8-46b8-b640-92d22665ca31" containerName="test-curl-7892-bb81f9" containerID="cri-o://d3864bc6d38f929c3b0ffb94f917468e9aa13af01666cfa0b4857ed1b44bfe80" gracePeriod=2
Sep 16 12:20:09.530936 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.530907    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"]
Sep 16 12:20:09.539708 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:20:09.539680    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2dd3b3_17b3_4556_9340_d4a69042ab8a.slice/crio-8801e8cce47a5ca5808734cb2add05946534661ffe50eae05427c181f78f3bf1 WatchSource:0}: Error finding container 8801e8cce47a5ca5808734cb2add05946534661ffe50eae05427c181f78f3bf1: Status 404 returned error can't find the container with id 8801e8cce47a5ca5808734cb2add05946534661ffe50eae05427c181f78f3bf1
Sep 16 12:20:09.542155 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.542120    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:09.542267 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.542207    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:09.542336 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.542279    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:09.648913 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.648878    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7892-73db19"
Sep 16 12:20:09.762835 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.762741    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"]
Sep 16 12:20:09.766833 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.766812    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.769440 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.769417    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5fb73413-openshift-default-dockercfg-vjq5d\""
Sep 16 12:20:09.776108 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.774041    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjwww\" (UniqueName: \"kubernetes.io/projected/9e0d3e92-11dc-4bdf-a34f-5b2e313f45f9-kube-api-access-zjwww\") pod \"9e0d3e92-11dc-4bdf-a34f-5b2e313f45f9\" (UID: \"9e0d3e92-11dc-4bdf-a34f-5b2e313f45f9\") "
Sep 16 12:20:09.776554 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.776474    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"]
Sep 16 12:20:09.777924 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.777102    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e0d3e92-11dc-4bdf-a34f-5b2e313f45f9-kube-api-access-zjwww" (OuterVolumeSpecName: "kube-api-access-zjwww") pod "9e0d3e92-11dc-4bdf-a34f-5b2e313f45f9" (UID: "9e0d3e92-11dc-4bdf-a34f-5b2e313f45f9"). InnerVolumeSpecName "kube-api-access-zjwww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:20:09.875179 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.875128    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istio-token\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.875179 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.875177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istiod-ca-cert\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.875426 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.875221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-workload-socket\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.875426 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.875267    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kcxk\" (UniqueName: \"kubernetes.io/projected/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-kube-api-access-8kcxk\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.875426 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.875323    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-workload-certs\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.875426 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.875369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istio-podinfo\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.875426 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.875401    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-credential-socket\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.875711 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.875460    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istio-envoy\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.875711 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.875485    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istio-data\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.875711 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.875562    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjwww\" (UniqueName: \"kubernetes.io/projected/9e0d3e92-11dc-4bdf-a34f-5b2e313f45f9-kube-api-access-zjwww\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:20:09.976406 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.976369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istio-podinfo\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.976406 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.976411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-credential-socket\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.976688 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.976449    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istio-envoy\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.976688 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.976465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istio-data\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.976688 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.976501    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istio-token\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.976688 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.976517    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istiod-ca-cert\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.976688 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.976544    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-workload-socket\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.976688 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.976574    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kcxk\" (UniqueName: \"kubernetes.io/projected/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-kube-api-access-8kcxk\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.976688 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.976596    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-workload-certs\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.977059 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.976990    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-credential-socket\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.977059 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.977016    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-workload-certs\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.977178 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.977069    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istio-data\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.977265 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.977244    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-workload-socket\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.977411 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.977382    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istiod-ca-cert\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.978985 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.978964    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istio-envoy\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.979354 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.979337    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istio-podinfo\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.984568 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.984547    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istio-token\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:09.984754 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:09.984738    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kcxk\" (UniqueName: \"kubernetes.io/projected/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-kube-api-access-8kcxk\") pod \"e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") " pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:10.082560 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:10.082532    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:10.370912 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:10.370875    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"]
Sep 16 12:20:10.372134 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:20:10.372106    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8610682_cd2c_4c74_b24e_fbcfb1d8e689.slice/crio-535c5af3f44fab9d00adf35e2236c511210726fa43546fce9ede35aa2d7fbc71 WatchSource:0}: Error finding container 535c5af3f44fab9d00adf35e2236c511210726fa43546fce9ede35aa2d7fbc71: Status 404 returned error can't find the container with id 535c5af3f44fab9d00adf35e2236c511210726fa43546fce9ede35aa2d7fbc71
Sep 16 12:20:10.374875 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:10.374831    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:10.374963 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:10.374923    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:10.375008 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:10.374967    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:10.522545 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:10.522494    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s" event={"ID":"c8610682-cd2c-4c74-b24e-fbcfb1d8e689","Type":"ContainerStarted","Data":"3f7cd23576ebddd7da1dcc3d2e3103e8ba75df409d0ec9be2f374ab0f76cfffc"}
Sep 16 12:20:10.523032 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:10.522558    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s" event={"ID":"c8610682-cd2c-4c74-b24e-fbcfb1d8e689","Type":"ContainerStarted","Data":"535c5af3f44fab9d00adf35e2236c511210726fa43546fce9ede35aa2d7fbc71"}
Sep 16 12:20:10.527735 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:10.524723    2579 generic.go:356] "Generic (PLEG): container finished" podID="9e0d3e92-11dc-4bdf-a34f-5b2e313f45f9" containerID="11261477603b413d354d9a75b0f547b60ecfbb465776de1174703ac527252180" exitCode=137
Sep 16 12:20:10.527735 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:10.524800    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7892-73db19"
Sep 16 12:20:10.527735 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:10.524827    2579 scope.go:117] "RemoveContainer" containerID="11261477603b413d354d9a75b0f547b60ecfbb465776de1174703ac527252180"
Sep 16 12:20:10.529117 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:10.529036    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw" event={"ID":"ae2dd3b3-17b3-4556-9340-d4a69042ab8a","Type":"ContainerStarted","Data":"3cd18fed7a719a18f3166d34d1b3c5f019b96441726967f3627146270fb2928c"}
Sep 16 12:20:10.529117 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:10.529109    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw" event={"ID":"ae2dd3b3-17b3-4556-9340-d4a69042ab8a","Type":"ContainerStarted","Data":"8801e8cce47a5ca5808734cb2add05946534661ffe50eae05427c181f78f3bf1"}
Sep 16 12:20:10.543677 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:10.542423    2579 scope.go:117] "RemoveContainer" containerID="11261477603b413d354d9a75b0f547b60ecfbb465776de1174703ac527252180"
Sep 16 12:20:10.543677 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:20:10.543561    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11261477603b413d354d9a75b0f547b60ecfbb465776de1174703ac527252180\": container with ID starting with 11261477603b413d354d9a75b0f547b60ecfbb465776de1174703ac527252180 not found: ID does not exist" containerID="11261477603b413d354d9a75b0f547b60ecfbb465776de1174703ac527252180"
Sep 16 12:20:10.543677 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:10.543604    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11261477603b413d354d9a75b0f547b60ecfbb465776de1174703ac527252180"} err="failed to get container status \"11261477603b413d354d9a75b0f547b60ecfbb465776de1174703ac527252180\": rpc error: code = NotFound desc = could not find container \"11261477603b413d354d9a75b0f547b60ecfbb465776de1174703ac527252180\": container with ID starting with 11261477603b413d354d9a75b0f547b60ecfbb465776de1174703ac527252180 not found: ID does not exist"
Sep 16 12:20:10.545520 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:10.545304    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s" podStartSLOduration=1.54528664 podStartE2EDuration="1.54528664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:10.542631097 +0000 UTC m=+1004.243702466" watchObservedRunningTime="2026-09-16 12:20:10.54528664 +0000 UTC m=+1004.246357967"
Sep 16 12:20:10.563172 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:20:10.563131    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7892-73db19\" is forbidden: User \"system:node:ip-10-0-137-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-30.ec2.internal' and this object" podUID="9e0d3e92-11dc-4bdf-a34f-5b2e313f45f9" pod="opendatahub/test-curl-7892-73db19"
Sep 16 12:20:10.563531 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:10.563480    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw" podStartSLOduration=2.563461874 podStartE2EDuration="2.563461874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:10.561138926 +0000 UTC m=+1004.262210240" watchObservedRunningTime="2026-09-16 12:20:10.563461874 +0000 UTC m=+1004.264533199"
Sep 16 12:20:10.782812 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:10.782724    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e0d3e92-11dc-4bdf-a34f-5b2e313f45f9" path="/var/lib/kubelet/pods/9e0d3e92-11dc-4bdf-a34f-5b2e313f45f9/volumes"
Sep 16 12:20:11.083095 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:11.083060    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:11.088330 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:11.088301    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:11.253182 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:11.253147    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:11.257916 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:11.257886    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:11.536179 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:11.536147    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:11.536557 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:11.536206    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:11.537077 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:11.537061    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:11.537487 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:11.537468    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:11.725779 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:11.725757    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7892-bb81f9"
Sep 16 12:20:11.796548 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:11.796452    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r68v6\" (UniqueName: \"kubernetes.io/projected/eb81778a-86b8-46b8-b640-92d22665ca31-kube-api-access-r68v6\") pod \"eb81778a-86b8-46b8-b640-92d22665ca31\" (UID: \"eb81778a-86b8-46b8-b640-92d22665ca31\") "
Sep 16 12:20:11.799352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:11.799305    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb81778a-86b8-46b8-b640-92d22665ca31-kube-api-access-r68v6" (OuterVolumeSpecName: "kube-api-access-r68v6") pod "eb81778a-86b8-46b8-b640-92d22665ca31" (UID: "eb81778a-86b8-46b8-b640-92d22665ca31"). InnerVolumeSpecName "kube-api-access-r68v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:20:11.897843 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:11.897804    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r68v6\" (UniqueName: \"kubernetes.io/projected/eb81778a-86b8-46b8-b640-92d22665ca31-kube-api-access-r68v6\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:20:12.547375 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:12.547328    2579 generic.go:356] "Generic (PLEG): container finished" podID="eb81778a-86b8-46b8-b640-92d22665ca31" containerID="d3864bc6d38f929c3b0ffb94f917468e9aa13af01666cfa0b4857ed1b44bfe80" exitCode=137
Sep 16 12:20:12.547375 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:12.547363    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7892-bb81f9"
Sep 16 12:20:12.547928 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:12.547425    2579 scope.go:117] "RemoveContainer" containerID="d3864bc6d38f929c3b0ffb94f917468e9aa13af01666cfa0b4857ed1b44bfe80"
Sep 16 12:20:12.564375 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:12.564336    2579 scope.go:117] "RemoveContainer" containerID="d3864bc6d38f929c3b0ffb94f917468e9aa13af01666cfa0b4857ed1b44bfe80"
Sep 16 12:20:12.565424 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:20:12.565307    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3864bc6d38f929c3b0ffb94f917468e9aa13af01666cfa0b4857ed1b44bfe80\": container with ID starting with d3864bc6d38f929c3b0ffb94f917468e9aa13af01666cfa0b4857ed1b44bfe80 not found: ID does not exist" containerID="d3864bc6d38f929c3b0ffb94f917468e9aa13af01666cfa0b4857ed1b44bfe80"
Sep 16 12:20:12.565424 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:12.565349    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3864bc6d38f929c3b0ffb94f917468e9aa13af01666cfa0b4857ed1b44bfe80"} err="failed to get container status \"d3864bc6d38f929c3b0ffb94f917468e9aa13af01666cfa0b4857ed1b44bfe80\": rpc error: code = NotFound desc = could not find container \"d3864bc6d38f929c3b0ffb94f917468e9aa13af01666cfa0b4857ed1b44bfe80\": container with ID starting with d3864bc6d38f929c3b0ffb94f917468e9aa13af01666cfa0b4857ed1b44bfe80 not found: ID does not exist"
Sep 16 12:20:12.567217 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:20:12.567176    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7892-bb81f9\" is forbidden: User \"system:node:ip-10-0-137-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-30.ec2.internal' and this object" podUID="eb81778a-86b8-46b8-b640-92d22665ca31" pod="opendatahub/test-curl-7892-bb81f9"
Sep 16 12:20:12.784509 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:12.784463    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb81778a-86b8-46b8-b640-92d22665ca31" path="/var/lib/kubelet/pods/eb81778a-86b8-46b8-b640-92d22665ca31/volumes"
Sep 16 12:20:18.746379 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.746343    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"]
Sep 16 12:20:18.751730 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.751701    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:18.754257 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.754233    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ef3317-openshift-default-dockercfg-s4h4n\""
Sep 16 12:20:18.770381 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.770352    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"]
Sep 16 12:20:18.877469 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.877433    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc5kp\" (UniqueName: \"kubernetes.io/projected/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-kube-api-access-tc5kp\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:18.877624 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.877484    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istiod-ca-cert\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:18.877624 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.877536    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-workload-certs\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:18.877624 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.877557    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-workload-socket\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:18.877846 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.877755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istio-podinfo\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:18.877846 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.877804    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istio-data\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:18.877846 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.877843    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istio-token\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:18.878007 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.877869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istio-envoy\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:18.878007 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.877908    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-credential-socket\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:18.978934 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.978883    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-workload-socket\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:18.979145 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.979015    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istio-podinfo\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:18.979145 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.979056    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istio-data\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:18.979145 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.979095    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istio-token\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:18.979145 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.979123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istio-envoy\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:18.979375 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.979167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-credential-socket\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:18.979375 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.979201    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc5kp\" (UniqueName: \"kubernetes.io/projected/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-kube-api-access-tc5kp\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:18.979375 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.979243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istiod-ca-cert\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:18.979375 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.979303    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-workload-certs\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:18.982677 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.979774    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-workload-certs\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:18.982677 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.980031    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-workload-socket\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:18.986675 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.982988    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istiod-ca-cert\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:18.986675 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.983107    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istio-podinfo\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:18.995160 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:18.995133    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istio-token\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:19.001718 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:19.000577    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istio-envoy\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:19.001718 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:19.000849    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-credential-socket\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:19.001718 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:19.000960    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istio-data\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:19.001718 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:19.001675    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc5kp\" (UniqueName: \"kubernetes.io/projected/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-kube-api-access-tc5kp\") pod \"e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") " pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:19.063045 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:19.063015    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:19.363211 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:19.363170    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"]
Sep 16 12:20:19.363965 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:20:19.363939    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2316ba71_eb2c_48d2_a6e2_cb2bd6bdf2c1.slice/crio-d48b43954a1e4025e980d6e37b391724896169f1b0f2b70caaab8c299f6623a0 WatchSource:0}: Error finding container d48b43954a1e4025e980d6e37b391724896169f1b0f2b70caaab8c299f6623a0: Status 404 returned error can't find the container with id d48b43954a1e4025e980d6e37b391724896169f1b0f2b70caaab8c299f6623a0
Sep 16 12:20:19.366460 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:19.366426    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:19.366545 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:19.366504    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:19.366602 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:19.366550    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:19.515156 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:19.514813    2579 patch_prober.go:28] interesting pod/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 12:20:19.515156 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:19.514880    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg" podUID="5ade63ee-3184-46df-ada2-55a290bcbefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 12:20:19.598734 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:19.598692    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7" event={"ID":"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1","Type":"ContainerStarted","Data":"9b14644d0cf41af7fcc4a9a4d4ea4db3e6641769c585586246def8ff6e393d00"}
Sep 16 12:20:19.598885 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:19.598766    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7" event={"ID":"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1","Type":"ContainerStarted","Data":"d48b43954a1e4025e980d6e37b391724896169f1b0f2b70caaab8c299f6623a0"}
Sep 16 12:20:19.625924 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:19.625854    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7" podStartSLOduration=1.625833335 podStartE2EDuration="1.625833335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:19.622327856 +0000 UTC m=+1013.323399217" watchObservedRunningTime="2026-09-16 12:20:19.625833335 +0000 UTC m=+1013.326904664"
Sep 16 12:20:20.063400 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:20.063302    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:20.069901 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:20.069869    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:20.271402 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:20.271356    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:20:20.606039 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:20.605993    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:20.607034 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:20.607009    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:22.621412 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:22.621376    2579 generic.go:356] "Generic (PLEG): container finished" podID="5ade63ee-3184-46df-ada2-55a290bcbefc" containerID="da5db95d12256e348e633bc22c6c62e9ece315dd0cb228c175d796fe6ac3f0b1" exitCode=0
Sep 16 12:20:22.621929 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:22.621451    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg" event={"ID":"5ade63ee-3184-46df-ada2-55a290bcbefc","Type":"ContainerDied","Data":"da5db95d12256e348e633bc22c6c62e9ece315dd0cb228c175d796fe6ac3f0b1"}
Sep 16 12:20:22.622108 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:22.622092    2579 scope.go:117] "RemoveContainer" containerID="da5db95d12256e348e633bc22c6c62e9ece315dd0cb228c175d796fe6ac3f0b1"
Sep 16 12:20:22.623170 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:22.623139    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:22.623296 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:22.623239    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:22.623296 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:22.623291    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:23.635321 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:23.635272    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg" event={"ID":"5ade63ee-3184-46df-ada2-55a290bcbefc","Type":"ContainerStarted","Data":"1077e9fbe6a5a07553e69cec535e3de9a5653d4b1be3440f71ea05b06a4ef89a"}
Sep 16 12:20:23.638606 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:23.638567    2579 generic.go:356] "Generic (PLEG): container finished" podID="f35bc2fb-f32f-4f0f-8e78-918f927616e3" containerID="255281c2fb0468af70a0473be5d68852a8712a74f640af6ef45ea760c676c037" exitCode=0
Sep 16 12:20:23.638782 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:23.638719    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx" event={"ID":"f35bc2fb-f32f-4f0f-8e78-918f927616e3","Type":"ContainerDied","Data":"255281c2fb0468af70a0473be5d68852a8712a74f640af6ef45ea760c676c037"}
Sep 16 12:20:23.639229 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:23.639205    2579 scope.go:117] "RemoveContainer" containerID="255281c2fb0468af70a0473be5d68852a8712a74f640af6ef45ea760c676c037"
Sep 16 12:20:23.640918 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:23.640879    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:23.641120 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:23.641091    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:23.641202 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:23.641160    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:24.238206 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:24.238159    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:24.238428 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:24.238328    2579 patch_prober.go:28] interesting pod/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 16 12:20:24.238428 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:24.238406    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg" podUID="5ade63ee-3184-46df-ada2-55a290bcbefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 16 12:20:24.656244 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:24.656202    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx" event={"ID":"f35bc2fb-f32f-4f0f-8e78-918f927616e3","Type":"ContainerStarted","Data":"9b701cefee6e2c48aec2a9a2f3d11ae125f4d01ffe043e25fdafa393df2fb446"}
Sep 16 12:20:25.075967 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:25.075413    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:25.075967 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:25.075548    2579 patch_prober.go:28] interesting pod/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 16 12:20:25.075967 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:25.075594    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx" podUID="f35bc2fb-f32f-4f0f-8e78-918f927616e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 16 12:20:25.237797 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:25.237756    2579 patch_prober.go:28] interesting pod/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 16 12:20:25.238130 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:25.238093    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg" podUID="5ade63ee-3184-46df-ada2-55a290bcbefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 16 12:20:26.075926 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:26.075892    2579 patch_prober.go:28] interesting pod/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 16 12:20:26.076887 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:26.076478    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx" podUID="f35bc2fb-f32f-4f0f-8e78-918f927616e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 16 12:20:26.237808 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:26.237344    2579 patch_prober.go:28] interesting pod/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 16 12:20:26.237808 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:26.237412    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg" podUID="5ade63ee-3184-46df-ada2-55a290bcbefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 16 12:20:27.075673 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:27.075547    2579 patch_prober.go:28] interesting pod/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 16 12:20:27.075673 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:27.075619    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx" podUID="f35bc2fb-f32f-4f0f-8e78-918f927616e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 16 12:20:27.248132 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:27.248085    2579 patch_prober.go:28] interesting pod/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 16 12:20:27.248612 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:27.248155    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg" podUID="5ade63ee-3184-46df-ada2-55a290bcbefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 16 12:20:28.075873 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:28.075778    2579 patch_prober.go:28] interesting pod/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 16 12:20:28.076130 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:28.076102    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx" podUID="f35bc2fb-f32f-4f0f-8e78-918f927616e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 16 12:20:28.237633 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:28.237594    2579 patch_prober.go:28] interesting pod/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 16 12:20:28.237982 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:28.237939    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg" podUID="5ade63ee-3184-46df-ada2-55a290bcbefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 16 12:20:29.075497 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:29.075451    2579 patch_prober.go:28] interesting pod/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 16 12:20:29.075875 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:29.075520    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx" podUID="f35bc2fb-f32f-4f0f-8e78-918f927616e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 16 12:20:29.237511 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:29.237232    2579 patch_prober.go:28] interesting pod/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 16 12:20:29.237511 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:29.237291    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg" podUID="5ade63ee-3184-46df-ada2-55a290bcbefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 16 12:20:30.075738 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:30.075692    2579 patch_prober.go:28] interesting pod/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 16 12:20:30.076235 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:30.075767    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx" podUID="f35bc2fb-f32f-4f0f-8e78-918f927616e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 16 12:20:30.241815 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:30.241782    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:30.242228 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:30.242198    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:30.243064 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:30.243046    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:20:31.075304 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:31.075265    2579 patch_prober.go:28] interesting pod/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Sep 16 12:20:31.075486 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:31.075345    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx" podUID="f35bc2fb-f32f-4f0f-8e78-918f927616e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Sep 16 12:20:32.080243 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:32.080201    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:32.082309 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:32.080613    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:32.082309 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:32.081526    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:20:32.310507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:32.310424    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fb73413-qggbx"]
Sep 16 12:20:32.313228 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:32.313197    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fb73413-qggbx"
Sep 16 12:20:32.315814 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:32.315789    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:20:32.315931 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:32.315817    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-767vd\""
Sep 16 12:20:32.321602 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:32.321572    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fb73413-qggbx"]
Sep 16 12:20:32.387680 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:32.385370    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ffb24b-76cc-42f9-abbd-d5193581eb44-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5fb73413-qggbx\" (UID: \"79ffb24b-76cc-42f9-abbd-d5193581eb44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fb73413-qggbx"
Sep 16 12:20:32.486293 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:32.486242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ffb24b-76cc-42f9-abbd-d5193581eb44-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5fb73413-qggbx\" (UID: \"79ffb24b-76cc-42f9-abbd-d5193581eb44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fb73413-qggbx"
Sep 16 12:20:32.487701 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:32.487676    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ffb24b-76cc-42f9-abbd-d5193581eb44-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5fb73413-qggbx\" (UID: \"79ffb24b-76cc-42f9-abbd-d5193581eb44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fb73413-qggbx"
Sep 16 12:20:32.629755 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:32.629718    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fb73413-qggbx"
Sep 16 12:20:32.738532 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:32.738367    2579 generic.go:356] "Generic (PLEG): container finished" podID="ae2dd3b3-17b3-4556-9340-d4a69042ab8a" containerID="3cd18fed7a719a18f3166d34d1b3c5f019b96441726967f3627146270fb2928c" exitCode=0
Sep 16 12:20:32.738532 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:32.738443    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw" event={"ID":"ae2dd3b3-17b3-4556-9340-d4a69042ab8a","Type":"ContainerDied","Data":"3cd18fed7a719a18f3166d34d1b3c5f019b96441726967f3627146270fb2928c"}
Sep 16 12:20:32.739725 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:32.738977    2579 scope.go:117] "RemoveContainer" containerID="3cd18fed7a719a18f3166d34d1b3c5f019b96441726967f3627146270fb2928c"
Sep 16 12:20:32.740466 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:32.740357    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:32.740466 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:32.740446    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:32.740608 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:32.740490    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:32.940065 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:32.940039    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fb73413-qggbx"]
Sep 16 12:20:32.941957 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:20:32.941931    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ffb24b_76cc_42f9_abbd_d5193581eb44.slice/crio-6e6bb7aedb2ae7691ce9464e1a1c61b717ec681d2dc41373d2f51c0e634d74c8 WatchSource:0}: Error finding container 6e6bb7aedb2ae7691ce9464e1a1c61b717ec681d2dc41373d2f51c0e634d74c8: Status 404 returned error can't find the container with id 6e6bb7aedb2ae7691ce9464e1a1c61b717ec681d2dc41373d2f51c0e634d74c8
Sep 16 12:20:33.748289 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:33.748246    2579 generic.go:356] "Generic (PLEG): container finished" podID="c8610682-cd2c-4c74-b24e-fbcfb1d8e689" containerID="3f7cd23576ebddd7da1dcc3d2e3103e8ba75df409d0ec9be2f374ab0f76cfffc" exitCode=0
Sep 16 12:20:33.748787 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:33.748376    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s" event={"ID":"c8610682-cd2c-4c74-b24e-fbcfb1d8e689","Type":"ContainerDied","Data":"3f7cd23576ebddd7da1dcc3d2e3103e8ba75df409d0ec9be2f374ab0f76cfffc"}
Sep 16 12:20:33.748920 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:33.748895    2579 scope.go:117] "RemoveContainer" containerID="3f7cd23576ebddd7da1dcc3d2e3103e8ba75df409d0ec9be2f374ab0f76cfffc"
Sep 16 12:20:33.750324 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:33.750287    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:33.750543 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:33.750512    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:33.750645 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:33.750577    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:33.750899 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:33.750878    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw" event={"ID":"ae2dd3b3-17b3-4556-9340-d4a69042ab8a","Type":"ContainerStarted","Data":"6be0d0cc8c814369d8c33ddaab732b4fea81c383f456e5c589617cd90f5414ee"}
Sep 16 12:20:33.752763 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:33.752415    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fb73413-qggbx" event={"ID":"79ffb24b-76cc-42f9-abbd-d5193581eb44","Type":"ContainerStarted","Data":"6e6bb7aedb2ae7691ce9464e1a1c61b717ec681d2dc41373d2f51c0e634d74c8"}
Sep 16 12:20:34.253227 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:34.253122    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:34.253454 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:34.253423    2579 patch_prober.go:28] interesting pod/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 16 12:20:34.253543 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:34.253492    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw" podUID="ae2dd3b3-17b3-4556-9340-d4a69042ab8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 16 12:20:34.771509 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:34.771465    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s" event={"ID":"c8610682-cd2c-4c74-b24e-fbcfb1d8e689","Type":"ContainerStarted","Data":"6146cae375701f81f87e6d03245ceb943c6a6257a19ca70b9b0f1b1eaa343590"}
Sep 16 12:20:35.083461 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:35.083423    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:35.083641 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:35.083552    2579 patch_prober.go:28] interesting pod/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 16 12:20:35.083641 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:35.083596    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s" podUID="c8610682-cd2c-4c74-b24e-fbcfb1d8e689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 16 12:20:35.253497 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:35.253001    2579 patch_prober.go:28] interesting pod/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 16 12:20:35.253497 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:35.253067    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw" podUID="ae2dd3b3-17b3-4556-9340-d4a69042ab8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 16 12:20:36.089505 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:36.089078    2579 patch_prober.go:28] interesting pod/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 16 12:20:36.089505 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:36.089150    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s" podUID="c8610682-cd2c-4c74-b24e-fbcfb1d8e689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 16 12:20:36.252951 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:36.252906    2579 patch_prober.go:28] interesting pod/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 16 12:20:36.253154 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:36.252977    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw" podUID="ae2dd3b3-17b3-4556-9340-d4a69042ab8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 16 12:20:37.086124 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:37.086086    2579 patch_prober.go:28] interesting pod/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 16 12:20:37.086416 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:37.086386    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s" podUID="c8610682-cd2c-4c74-b24e-fbcfb1d8e689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 16 12:20:37.254873 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:37.254361    2579 patch_prober.go:28] interesting pod/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 16 12:20:37.254873 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:37.254427    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw" podUID="ae2dd3b3-17b3-4556-9340-d4a69042ab8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 16 12:20:38.093240 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:38.093194    2579 patch_prober.go:28] interesting pod/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 16 12:20:38.093432 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:38.093271    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s" podUID="c8610682-cd2c-4c74-b24e-fbcfb1d8e689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 16 12:20:38.252587 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:38.252557    2579 patch_prober.go:28] interesting pod/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 16 12:20:38.252855 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:38.252829    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw" podUID="ae2dd3b3-17b3-4556-9340-d4a69042ab8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 16 12:20:38.821474 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:38.821366    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fb73413-qggbx" event={"ID":"79ffb24b-76cc-42f9-abbd-d5193581eb44","Type":"ContainerStarted","Data":"19afd3b3595cd4df0f9978f897ca3077edb04ef0eb54b065994a8f53ac5744f6"}
Sep 16 12:20:38.839879 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:38.839815    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fb73413-qggbx" podStartSLOduration=1.676637528 podStartE2EDuration="6.839796308s" totalImagesPullingTime="5.16315878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:38.839795738 +0000 UTC m=+1032.540867064" watchObservedRunningTime="2026-09-16 12:20:38.839796308 +0000 UTC m=+1032.540867635"
Sep 16 12:20:39.083722 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:39.083681    2579 patch_prober.go:28] interesting pod/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 16 12:20:39.083914 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:39.083758    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s" podUID="c8610682-cd2c-4c74-b24e-fbcfb1d8e689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 16 12:20:39.253494 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:39.253456    2579 patch_prober.go:28] interesting pod/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Sep 16 12:20:39.253683 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:39.253527    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw" podUID="ae2dd3b3-17b3-4556-9340-d4a69042ab8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Sep 16 12:20:39.829395 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:39.829360    2579 generic.go:356] "Generic (PLEG): container finished" podID="79ffb24b-76cc-42f9-abbd-d5193581eb44" containerID="19afd3b3595cd4df0f9978f897ca3077edb04ef0eb54b065994a8f53ac5744f6" exitCode=0
Sep 16 12:20:39.829801 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:39.829457    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fb73413-qggbx" event={"ID":"79ffb24b-76cc-42f9-abbd-d5193581eb44","Type":"ContainerDied","Data":"19afd3b3595cd4df0f9978f897ca3077edb04ef0eb54b065994a8f53ac5744f6"}
Sep 16 12:20:39.908568 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:39.908524    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:20:40.083767 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:40.083724    2579 patch_prober.go:28] interesting pod/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 16 12:20:40.083974 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:40.083800    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s" podUID="c8610682-cd2c-4c74-b24e-fbcfb1d8e689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 16 12:20:40.258528 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:40.258492    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:40.258895 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:40.258867    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:40.259607 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:40.259585    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:20:40.971336 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:40.971310    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fb73413-qggbx"
Sep 16 12:20:41.057915 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:41.057867    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ffb24b-76cc-42f9-abbd-d5193581eb44-maas-api-service-ca\") pod \"79ffb24b-76cc-42f9-abbd-d5193581eb44\" (UID: \"79ffb24b-76cc-42f9-abbd-d5193581eb44\") "
Sep 16 12:20:41.058256 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:41.058228    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ffb24b-76cc-42f9-abbd-d5193581eb44-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "79ffb24b-76cc-42f9-abbd-d5193581eb44" (UID: "79ffb24b-76cc-42f9-abbd-d5193581eb44"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:20:41.087148 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:41.087062    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:41.087383 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:41.087355    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:41.088102 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:41.088074    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:41.159187 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:41.159142    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ffb24b-76cc-42f9-abbd-d5193581eb44-maas-api-service-ca\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:20:41.844971 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:41.844925    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fb73413-qggbx" event={"ID":"79ffb24b-76cc-42f9-abbd-d5193581eb44","Type":"ContainerDied","Data":"6e6bb7aedb2ae7691ce9464e1a1c61b717ec681d2dc41373d2f51c0e634d74c8"}
Sep 16 12:20:41.844971 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:41.844953    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fb73413-qggbx"
Sep 16 12:20:41.845268 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:41.844989    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e6bb7aedb2ae7691ce9464e1a1c61b717ec681d2dc41373d2f51c0e634d74c8"
Sep 16 12:20:42.851984 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:42.851949    2579 generic.go:356] "Generic (PLEG): container finished" podID="2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1" containerID="9b14644d0cf41af7fcc4a9a4d4ea4db3e6641769c585586246def8ff6e393d00" exitCode=0
Sep 16 12:20:42.852405 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:42.852020    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7" event={"ID":"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1","Type":"ContainerDied","Data":"9b14644d0cf41af7fcc4a9a4d4ea4db3e6641769c585586246def8ff6e393d00"}
Sep 16 12:20:42.852568 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:42.852550    2579 scope.go:117] "RemoveContainer" containerID="9b14644d0cf41af7fcc4a9a4d4ea4db3e6641769c585586246def8ff6e393d00"
Sep 16 12:20:42.853666 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:42.853626    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:42.853742 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:42.853726    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:42.853780 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:42.853765    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:20:43.859729 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:43.859686    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7" event={"ID":"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1","Type":"ContainerStarted","Data":"d4025312c43c98c1c39d1dd55e577a8879ad252f026aa1a5c0d066d9c307d25e"}
Sep 16 12:20:44.063826 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:44.063790    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:44.064031 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:44.063981    2579 patch_prober.go:28] interesting pod/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 16 12:20:44.064107 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:44.064058    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7" podUID="2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 16 12:20:44.162786 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:44.162704    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fb73413-qggbx"]
Sep 16 12:20:44.165174 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:44.165148    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5fb73413-qggbx"]
Sep 16 12:20:44.304682 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:44.302374    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"]
Sep 16 12:20:44.304682 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:44.302629    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s" podUID="c8610682-cd2c-4c74-b24e-fbcfb1d8e689" containerName="istio-proxy" containerID="cri-o://6146cae375701f81f87e6d03245ceb943c6a6257a19ca70b9b0f1b1eaa343590" gracePeriod=30
Sep 16 12:20:44.782849 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:44.782806    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79ffb24b-76cc-42f9-abbd-d5193581eb44" path="/var/lib/kubelet/pods/79ffb24b-76cc-42f9-abbd-d5193581eb44/volumes"
Sep 16 12:20:45.063730 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:45.063621    2579 patch_prober.go:28] interesting pod/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 16 12:20:45.063730 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:45.063690    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7" podUID="2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 16 12:20:46.063854 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:46.063819    2579 patch_prober.go:28] interesting pod/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 16 12:20:46.064243 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:46.063877    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7" podUID="2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 16 12:20:47.064062 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:47.064027    2579 patch_prober.go:28] interesting pod/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 16 12:20:47.064436 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:47.064085    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7" podUID="2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 16 12:20:48.064275 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:48.064242    2579 patch_prober.go:28] interesting pod/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 16 12:20:48.064695 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:48.064298    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7" podUID="2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 16 12:20:49.067327 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.067294    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:49.067777 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.067602    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:49.068157 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.068141    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:20:49.440110 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.440073    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g"]
Sep 16 12:20:49.440493 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.440480    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79ffb24b-76cc-42f9-abbd-d5193581eb44" containerName="revoke-keys"
Sep 16 12:20:49.440539 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.440495    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="79ffb24b-76cc-42f9-abbd-d5193581eb44" containerName="revoke-keys"
Sep 16 12:20:49.440588 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.440575    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="79ffb24b-76cc-42f9-abbd-d5193581eb44" containerName="revoke-keys"
Sep 16 12:20:49.442949 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.442929    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g"
Sep 16 12:20:49.445422 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.445395    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-4c69c87e-serving-cert\""
Sep 16 12:20:49.445574 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.445427    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 12:20:49.445574 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.445474    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:20:49.445574 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.445506    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 12:20:49.445574 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.445399    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4gkmq\""
Sep 16 12:20:49.452590 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.452288    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj68q\" (UniqueName: \"kubernetes.io/projected/85e8cf84-fdac-42fd-a568-99c88f0ae407-kube-api-access-xj68q\") pod \"maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g\" (UID: \"85e8cf84-fdac-42fd-a568-99c88f0ae407\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g"
Sep 16 12:20:49.452590 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.452376    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85e8cf84-fdac-42fd-a568-99c88f0ae407-openshift-service-ca\") pod \"maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g\" (UID: \"85e8cf84-fdac-42fd-a568-99c88f0ae407\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g"
Sep 16 12:20:49.452590 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.452468    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e8cf84-fdac-42fd-a568-99c88f0ae407-metrics-certs\") pod \"maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g\" (UID: \"85e8cf84-fdac-42fd-a568-99c88f0ae407\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g"
Sep 16 12:20:49.452590 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.452505    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85e8cf84-fdac-42fd-a568-99c88f0ae407-maas-api-tls\") pod \"maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g\" (UID: \"85e8cf84-fdac-42fd-a568-99c88f0ae407\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g"
Sep 16 12:20:49.453441 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.453116    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g"]
Sep 16 12:20:49.553772 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.553740    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e8cf84-fdac-42fd-a568-99c88f0ae407-metrics-certs\") pod \"maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g\" (UID: \"85e8cf84-fdac-42fd-a568-99c88f0ae407\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g"
Sep 16 12:20:49.553967 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.553790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85e8cf84-fdac-42fd-a568-99c88f0ae407-maas-api-tls\") pod \"maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g\" (UID: \"85e8cf84-fdac-42fd-a568-99c88f0ae407\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g"
Sep 16 12:20:49.553967 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.553849    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj68q\" (UniqueName: \"kubernetes.io/projected/85e8cf84-fdac-42fd-a568-99c88f0ae407-kube-api-access-xj68q\") pod \"maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g\" (UID: \"85e8cf84-fdac-42fd-a568-99c88f0ae407\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g"
Sep 16 12:20:49.553967 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.553921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85e8cf84-fdac-42fd-a568-99c88f0ae407-openshift-service-ca\") pod \"maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g\" (UID: \"85e8cf84-fdac-42fd-a568-99c88f0ae407\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g"
Sep 16 12:20:49.554628 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.554602    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85e8cf84-fdac-42fd-a568-99c88f0ae407-openshift-service-ca\") pod \"maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g\" (UID: \"85e8cf84-fdac-42fd-a568-99c88f0ae407\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g"
Sep 16 12:20:49.556299 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.556276    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85e8cf84-fdac-42fd-a568-99c88f0ae407-maas-api-tls\") pod \"maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g\" (UID: \"85e8cf84-fdac-42fd-a568-99c88f0ae407\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g"
Sep 16 12:20:49.556565 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.556544    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e8cf84-fdac-42fd-a568-99c88f0ae407-metrics-certs\") pod \"maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g\" (UID: \"85e8cf84-fdac-42fd-a568-99c88f0ae407\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g"
Sep 16 12:20:49.561640 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.561618    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj68q\" (UniqueName: \"kubernetes.io/projected/85e8cf84-fdac-42fd-a568-99c88f0ae407-kube-api-access-xj68q\") pod \"maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g\" (UID: \"85e8cf84-fdac-42fd-a568-99c88f0ae407\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g"
Sep 16 12:20:49.669590 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.669563    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:49.756192 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.756101    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istio-podinfo\") pod \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") "
Sep 16 12:20:49.756192 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.756180    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-credential-socket\") pod \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") "
Sep 16 12:20:49.756420 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.756221    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-workload-certs\") pod \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") "
Sep 16 12:20:49.756420 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.756245    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istio-token\") pod \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") "
Sep 16 12:20:49.756420 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.756269    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-workload-socket\") pod \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") "
Sep 16 12:20:49.756420 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.756299    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istio-envoy\") pod \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") "
Sep 16 12:20:49.756420 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.756322    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istio-data\") pod \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") "
Sep 16 12:20:49.756420 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.756359    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kcxk\" (UniqueName: \"kubernetes.io/projected/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-kube-api-access-8kcxk\") pod \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") "
Sep 16 12:20:49.756420 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.756386    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istiod-ca-cert\") pod \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\" (UID: \"c8610682-cd2c-4c74-b24e-fbcfb1d8e689\") "
Sep 16 12:20:49.756835 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.756514    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c8610682-cd2c-4c74-b24e-fbcfb1d8e689" (UID: "c8610682-cd2c-4c74-b24e-fbcfb1d8e689"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:20:49.756835 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.756534    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c8610682-cd2c-4c74-b24e-fbcfb1d8e689" (UID: "c8610682-cd2c-4c74-b24e-fbcfb1d8e689"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:20:49.756835 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.756644    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istio-data" (OuterVolumeSpecName: "istio-data") pod "c8610682-cd2c-4c74-b24e-fbcfb1d8e689" (UID: "c8610682-cd2c-4c74-b24e-fbcfb1d8e689"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:20:49.756835 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.756682    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c8610682-cd2c-4c74-b24e-fbcfb1d8e689" (UID: "c8610682-cd2c-4c74-b24e-fbcfb1d8e689"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:20:49.757051 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.756835    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g"
Sep 16 12:20:49.757051 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.756864    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c8610682-cd2c-4c74-b24e-fbcfb1d8e689" (UID: "c8610682-cd2c-4c74-b24e-fbcfb1d8e689"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:20:49.757051 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.756884    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-credential-socket\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:20:49.757051 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.756898    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-workload-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:20:49.757051 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.756908    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-workload-socket\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:20:49.757051 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.756915    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istio-data\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:20:49.758970 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.758920    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c8610682-cd2c-4c74-b24e-fbcfb1d8e689" (UID: "c8610682-cd2c-4c74-b24e-fbcfb1d8e689"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:20:49.759043 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.759009    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-kube-api-access-8kcxk" (OuterVolumeSpecName: "kube-api-access-8kcxk") pod "c8610682-cd2c-4c74-b24e-fbcfb1d8e689" (UID: "c8610682-cd2c-4c74-b24e-fbcfb1d8e689"). InnerVolumeSpecName "kube-api-access-8kcxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:20:49.759043 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.759022    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c8610682-cd2c-4c74-b24e-fbcfb1d8e689" (UID: "c8610682-cd2c-4c74-b24e-fbcfb1d8e689"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:20:49.759126 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.759039    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istio-token" (OuterVolumeSpecName: "istio-token") pod "c8610682-cd2c-4c74-b24e-fbcfb1d8e689" (UID: "c8610682-cd2c-4c74-b24e-fbcfb1d8e689"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:20:49.858184 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.858150    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istio-envoy\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:20:49.858184 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.858182    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kcxk\" (UniqueName: \"kubernetes.io/projected/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-kube-api-access-8kcxk\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:20:49.858370 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.858192    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istiod-ca-cert\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:20:49.858370 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.858200    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istio-podinfo\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:20:49.858370 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.858208    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8610682-cd2c-4c74-b24e-fbcfb1d8e689-istio-token\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:20:49.902704 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.902652    2579 generic.go:356] "Generic (PLEG): container finished" podID="c8610682-cd2c-4c74-b24e-fbcfb1d8e689" containerID="6146cae375701f81f87e6d03245ceb943c6a6257a19ca70b9b0f1b1eaa343590" exitCode=0
Sep 16 12:20:49.902854 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.902700    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s" event={"ID":"c8610682-cd2c-4c74-b24e-fbcfb1d8e689","Type":"ContainerDied","Data":"6146cae375701f81f87e6d03245ceb943c6a6257a19ca70b9b0f1b1eaa343590"}
Sep 16 12:20:49.902854 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.902729    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"
Sep 16 12:20:49.902854 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.902768    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s" event={"ID":"c8610682-cd2c-4c74-b24e-fbcfb1d8e689","Type":"ContainerDied","Data":"535c5af3f44fab9d00adf35e2236c511210726fa43546fce9ede35aa2d7fbc71"}
Sep 16 12:20:49.902854 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.902801    2579 scope.go:117] "RemoveContainer" containerID="6146cae375701f81f87e6d03245ceb943c6a6257a19ca70b9b0f1b1eaa343590"
Sep 16 12:20:49.914321 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.913874    2579 scope.go:117] "RemoveContainer" containerID="3f7cd23576ebddd7da1dcc3d2e3103e8ba75df409d0ec9be2f374ab0f76cfffc"
Sep 16 12:20:49.923377 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.923359    2579 scope.go:117] "RemoveContainer" containerID="6146cae375701f81f87e6d03245ceb943c6a6257a19ca70b9b0f1b1eaa343590"
Sep 16 12:20:49.923683 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:20:49.923651    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6146cae375701f81f87e6d03245ceb943c6a6257a19ca70b9b0f1b1eaa343590\": container with ID starting with 6146cae375701f81f87e6d03245ceb943c6a6257a19ca70b9b0f1b1eaa343590 not found: ID does not exist" containerID="6146cae375701f81f87e6d03245ceb943c6a6257a19ca70b9b0f1b1eaa343590"
Sep 16 12:20:49.923748 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.923691    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6146cae375701f81f87e6d03245ceb943c6a6257a19ca70b9b0f1b1eaa343590"} err="failed to get container status \"6146cae375701f81f87e6d03245ceb943c6a6257a19ca70b9b0f1b1eaa343590\": rpc error: code = NotFound desc = could not find container \"6146cae375701f81f87e6d03245ceb943c6a6257a19ca70b9b0f1b1eaa343590\": container with ID starting with 6146cae375701f81f87e6d03245ceb943c6a6257a19ca70b9b0f1b1eaa343590 not found: ID does not exist"
Sep 16 12:20:49.923748 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.923708    2579 scope.go:117] "RemoveContainer" containerID="3f7cd23576ebddd7da1dcc3d2e3103e8ba75df409d0ec9be2f374ab0f76cfffc"
Sep 16 12:20:49.924017 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:20:49.923988    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f7cd23576ebddd7da1dcc3d2e3103e8ba75df409d0ec9be2f374ab0f76cfffc\": container with ID starting with 3f7cd23576ebddd7da1dcc3d2e3103e8ba75df409d0ec9be2f374ab0f76cfffc not found: ID does not exist" containerID="3f7cd23576ebddd7da1dcc3d2e3103e8ba75df409d0ec9be2f374ab0f76cfffc"
Sep 16 12:20:49.924088 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.924027    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7cd23576ebddd7da1dcc3d2e3103e8ba75df409d0ec9be2f374ab0f76cfffc"} err="failed to get container status \"3f7cd23576ebddd7da1dcc3d2e3103e8ba75df409d0ec9be2f374ab0f76cfffc\": rpc error: code = NotFound desc = could not find container \"3f7cd23576ebddd7da1dcc3d2e3103e8ba75df409d0ec9be2f374ab0f76cfffc\": container with ID starting with 3f7cd23576ebddd7da1dcc3d2e3103e8ba75df409d0ec9be2f374ab0f76cfffc not found: ID does not exist"
Sep 16 12:20:49.933334 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.933305    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"]
Sep 16 12:20:49.937344 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:49.937316    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5fb73413-openshift-default-5f6c6f8bfd-q449s"]
Sep 16 12:20:50.036286 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:50.036261    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g"]
Sep 16 12:20:50.037902 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:20:50.037874    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85e8cf84_fdac_42fd_a568_99c88f0ae407.slice/crio-f5939b4e9e36ab01ec9b6f02fda2e7191212629590479bf7de0e4000a2f5c04b WatchSource:0}: Error finding container f5939b4e9e36ab01ec9b6f02fda2e7191212629590479bf7de0e4000a2f5c04b: Status 404 returned error can't find the container with id f5939b4e9e36ab01ec9b6f02fda2e7191212629590479bf7de0e4000a2f5c04b
Sep 16 12:20:50.792218 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:50.792134    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8610682-cd2c-4c74-b24e-fbcfb1d8e689" path="/var/lib/kubelet/pods/c8610682-cd2c-4c74-b24e-fbcfb1d8e689/volumes"
Sep 16 12:20:50.875755 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:50.875722    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:20:50.911916 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:50.911866    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g" event={"ID":"85e8cf84-fdac-42fd-a568-99c88f0ae407","Type":"ContainerStarted","Data":"f5939b4e9e36ab01ec9b6f02fda2e7191212629590479bf7de0e4000a2f5c04b"}
Sep 16 12:20:52.927259 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:52.927220    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g" event={"ID":"85e8cf84-fdac-42fd-a568-99c88f0ae407","Type":"ContainerStarted","Data":"6c2f22430ab604003a96a0d80f805bd8352ea15e25cf9660c9c53b3561179ba3"}
Sep 16 12:20:52.927768 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:52.927325    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g"
Sep 16 12:20:52.944910 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:52.944858    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g" podStartSLOduration=1.5456206620000001 podStartE2EDuration="3.944841256s" totalImagesPullingTime="2.399220594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:20:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:20:52.942629283 +0000 UTC m=+1046.643700623" watchObservedRunningTime="2026-09-16 12:20:52.944841256 +0000 UTC m=+1046.645912608"
Sep 16 12:20:58.939332 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:20:58.939295    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g"
Sep 16 12:21:07.567383 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:07.567347    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:21:28.736224 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:28.736189    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g"]
Sep 16 12:21:28.736730 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:28.736421    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g" podUID="85e8cf84-fdac-42fd-a568-99c88f0ae407" containerName="maas-api" containerID="cri-o://6c2f22430ab604003a96a0d80f805bd8352ea15e25cf9660c9c53b3561179ba3" gracePeriod=30
Sep 16 12:21:28.993074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:28.992938    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g"
Sep 16 12:21:29.044450 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.044405    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xj68q\" (UniqueName: \"kubernetes.io/projected/85e8cf84-fdac-42fd-a568-99c88f0ae407-kube-api-access-xj68q\") pod \"85e8cf84-fdac-42fd-a568-99c88f0ae407\" (UID: \"85e8cf84-fdac-42fd-a568-99c88f0ae407\") "
Sep 16 12:21:29.044629 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.044458    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e8cf84-fdac-42fd-a568-99c88f0ae407-metrics-certs\") pod \"85e8cf84-fdac-42fd-a568-99c88f0ae407\" (UID: \"85e8cf84-fdac-42fd-a568-99c88f0ae407\") "
Sep 16 12:21:29.044629 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.044502    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85e8cf84-fdac-42fd-a568-99c88f0ae407-openshift-service-ca\") pod \"85e8cf84-fdac-42fd-a568-99c88f0ae407\" (UID: \"85e8cf84-fdac-42fd-a568-99c88f0ae407\") "
Sep 16 12:21:29.044629 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.044585    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85e8cf84-fdac-42fd-a568-99c88f0ae407-maas-api-tls\") pod \"85e8cf84-fdac-42fd-a568-99c88f0ae407\" (UID: \"85e8cf84-fdac-42fd-a568-99c88f0ae407\") "
Sep 16 12:21:29.045135 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.045106    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85e8cf84-fdac-42fd-a568-99c88f0ae407-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "85e8cf84-fdac-42fd-a568-99c88f0ae407" (UID: "85e8cf84-fdac-42fd-a568-99c88f0ae407"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:21:29.045348 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.045324    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85e8cf84-fdac-42fd-a568-99c88f0ae407-openshift-service-ca\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:21:29.048544 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.048509    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e8cf84-fdac-42fd-a568-99c88f0ae407-kube-api-access-xj68q" (OuterVolumeSpecName: "kube-api-access-xj68q") pod "85e8cf84-fdac-42fd-a568-99c88f0ae407" (UID: "85e8cf84-fdac-42fd-a568-99c88f0ae407"). InnerVolumeSpecName "kube-api-access-xj68q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:21:29.050541 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.050518    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85e8cf84-fdac-42fd-a568-99c88f0ae407-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "85e8cf84-fdac-42fd-a568-99c88f0ae407" (UID: "85e8cf84-fdac-42fd-a568-99c88f0ae407"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:21:29.050781 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.050756    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85e8cf84-fdac-42fd-a568-99c88f0ae407-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "85e8cf84-fdac-42fd-a568-99c88f0ae407" (UID: "85e8cf84-fdac-42fd-a568-99c88f0ae407"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:21:29.163511 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.163473    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj68q\" (UniqueName: \"kubernetes.io/projected/85e8cf84-fdac-42fd-a568-99c88f0ae407-kube-api-access-xj68q\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:21:29.163785 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.163767    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85e8cf84-fdac-42fd-a568-99c88f0ae407-metrics-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:21:29.163905 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.163892    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85e8cf84-fdac-42fd-a568-99c88f0ae407-maas-api-tls\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:21:29.164725 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.164689    2579 generic.go:356] "Generic (PLEG): container finished" podID="85e8cf84-fdac-42fd-a568-99c88f0ae407" containerID="6c2f22430ab604003a96a0d80f805bd8352ea15e25cf9660c9c53b3561179ba3" exitCode=0
Sep 16 12:21:29.164827 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.164735    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g"
Sep 16 12:21:29.164827 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.164768    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g" event={"ID":"85e8cf84-fdac-42fd-a568-99c88f0ae407","Type":"ContainerDied","Data":"6c2f22430ab604003a96a0d80f805bd8352ea15e25cf9660c9c53b3561179ba3"}
Sep 16 12:21:29.164912 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.164829    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g" event={"ID":"85e8cf84-fdac-42fd-a568-99c88f0ae407","Type":"ContainerDied","Data":"f5939b4e9e36ab01ec9b6f02fda2e7191212629590479bf7de0e4000a2f5c04b"}
Sep 16 12:21:29.164912 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.164857    2579 scope.go:117] "RemoveContainer" containerID="6c2f22430ab604003a96a0d80f805bd8352ea15e25cf9660c9c53b3561179ba3"
Sep 16 12:21:29.175350 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.175331    2579 scope.go:117] "RemoveContainer" containerID="6c2f22430ab604003a96a0d80f805bd8352ea15e25cf9660c9c53b3561179ba3"
Sep 16 12:21:29.175631 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:21:29.175611    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c2f22430ab604003a96a0d80f805bd8352ea15e25cf9660c9c53b3561179ba3\": container with ID starting with 6c2f22430ab604003a96a0d80f805bd8352ea15e25cf9660c9c53b3561179ba3 not found: ID does not exist" containerID="6c2f22430ab604003a96a0d80f805bd8352ea15e25cf9660c9c53b3561179ba3"
Sep 16 12:21:29.175706 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.175640    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c2f22430ab604003a96a0d80f805bd8352ea15e25cf9660c9c53b3561179ba3"} err="failed to get container status \"6c2f22430ab604003a96a0d80f805bd8352ea15e25cf9660c9c53b3561179ba3\": rpc error: code = NotFound desc = could not find container \"6c2f22430ab604003a96a0d80f805bd8352ea15e25cf9660c9c53b3561179ba3\": container with ID starting with 6c2f22430ab604003a96a0d80f805bd8352ea15e25cf9660c9c53b3561179ba3 not found: ID does not exist"
Sep 16 12:21:29.186854 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.186830    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g"]
Sep 16 12:21:29.189305 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.189280    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g"]
Sep 16 12:21:29.372119 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.372067    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:21:29.782626 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.782544    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"]
Sep 16 12:21:29.783190 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.783171    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8610682-cd2c-4c74-b24e-fbcfb1d8e689" containerName="istio-proxy"
Sep 16 12:21:29.783190 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.783190    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8610682-cd2c-4c74-b24e-fbcfb1d8e689" containerName="istio-proxy"
Sep 16 12:21:29.783352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.783212    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85e8cf84-fdac-42fd-a568-99c88f0ae407" containerName="maas-api"
Sep 16 12:21:29.783352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.783222    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="85e8cf84-fdac-42fd-a568-99c88f0ae407" containerName="maas-api"
Sep 16 12:21:29.783352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.783249    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8610682-cd2c-4c74-b24e-fbcfb1d8e689" containerName="istio-proxy"
Sep 16 12:21:29.783352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.783257    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8610682-cd2c-4c74-b24e-fbcfb1d8e689" containerName="istio-proxy"
Sep 16 12:21:29.783352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.783341    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8610682-cd2c-4c74-b24e-fbcfb1d8e689" containerName="istio-proxy"
Sep 16 12:21:29.783352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.783352    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="85e8cf84-fdac-42fd-a568-99c88f0ae407" containerName="maas-api"
Sep 16 12:21:29.783544 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.783362    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8610682-cd2c-4c74-b24e-fbcfb1d8e689" containerName="istio-proxy"
Sep 16 12:21:29.786505 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.786475    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.789102 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.789081    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1f9d3de8-openshift-default-dockercfg-h9qbl\""
Sep 16 12:21:29.793559 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.793536    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"]
Sep 16 12:21:29.868989 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.868955    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istio-podinfo\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.869288 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.869001    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istio-token\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.869288 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.869024    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istio-data\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.869288 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.869118    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq9j7\" (UniqueName: \"kubernetes.io/projected/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-kube-api-access-hq9j7\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.869288 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.869174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-workload-certs\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.869288 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.869211    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-workload-socket\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.869288 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.869253    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istio-envoy\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.869509 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.869301    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-credential-socket\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.869509 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.869343    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istiod-ca-cert\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.970084 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.970051    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istiod-ca-cert\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.970257 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.970104    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istio-podinfo\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.970257 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.970132    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istio-token\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.970257 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.970159    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istio-data\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.970257 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.970188    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq9j7\" (UniqueName: \"kubernetes.io/projected/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-kube-api-access-hq9j7\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.970257 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.970220    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-workload-certs\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.970257 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.970251    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-workload-socket\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.970571 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.970380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istio-envoy\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.970571 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.970444    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-credential-socket\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.970701 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.970568    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istio-data\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.970701 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.970600    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-workload-certs\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.970701 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.970648    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-workload-socket\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.970855 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.970825    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istiod-ca-cert\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.970893 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.970847    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-credential-socket\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.972586 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.972557    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istio-envoy\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.972762 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.972737    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istio-podinfo\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.977986 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.977961    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istio-token\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:29.978352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:29.978325    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq9j7\" (UniqueName: \"kubernetes.io/projected/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-kube-api-access-hq9j7\") pod \"e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") " pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:30.099005 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:30.098970    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:30.376799 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:30.376772    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"]
Sep 16 12:21:30.378086 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:21:30.378050    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8af3a6_2d34_4b3f_ab9f_676c8450ab22.slice/crio-1d40491f7e1ffc8ffd331896a55665b244106c5d168ec717861b34862d7b1ab3 WatchSource:0}: Error finding container 1d40491f7e1ffc8ffd331896a55665b244106c5d168ec717861b34862d7b1ab3: Status 404 returned error can't find the container with id 1d40491f7e1ffc8ffd331896a55665b244106c5d168ec717861b34862d7b1ab3
Sep 16 12:21:30.380630 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:30.380594    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:21:30.380731 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:30.380717    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:21:30.380773 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:30.380754    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:21:30.781511 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:30.781431    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85e8cf84-fdac-42fd-a568-99c88f0ae407" path="/var/lib/kubelet/pods/85e8cf84-fdac-42fd-a568-99c88f0ae407/volumes"
Sep 16 12:21:31.179425 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:31.179384    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p" event={"ID":"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22","Type":"ContainerStarted","Data":"574d069cf6ea231c2135b0159a5bcfdce69c8f5ffc8e3684609ec95c8c5af066"}
Sep 16 12:21:31.179865 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:31.179459    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p" event={"ID":"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22","Type":"ContainerStarted","Data":"1d40491f7e1ffc8ffd331896a55665b244106c5d168ec717861b34862d7b1ab3"}
Sep 16 12:21:31.199297 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:31.199220    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p" podStartSLOduration=2.199202648 podStartE2EDuration="2.199202648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:21:31.198344302 +0000 UTC m=+1084.899415656" watchObservedRunningTime="2026-09-16 12:21:31.199202648 +0000 UTC m=+1084.900274054"
Sep 16 12:21:31.934971 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:31.934931    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4c69c87e-56f559fb7c-lmv6g" podUID="85e8cf84-fdac-42fd-a568-99c88f0ae407" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.53:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 12:21:32.099767 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:32.099733    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:32.104306 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:32.104284    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:32.187034 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:32.186942    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:32.187865 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:32.187848    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:34.117422 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:34.117388    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp"]
Sep 16 12:21:34.121010 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:34.120990    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp"
Sep 16 12:21:34.125129 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:34.125100    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4gkmq\""
Sep 16 12:21:34.125258 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:34.125161    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:21:34.125258 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:34.125167    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 12:21:34.125385 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:34.125279    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1f9d3de8-serving-cert\""
Sep 16 12:21:34.125442 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:34.125398    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 12:21:34.132934 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:34.132908    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp"]
Sep 16 12:21:34.211443 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:34.211410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f0e5059-20d5-4772-afce-65a0c9f27c3a-openshift-service-ca\") pod \"maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp\" (UID: \"3f0e5059-20d5-4772-afce-65a0c9f27c3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp"
Sep 16 12:21:34.211443 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:34.211441    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xxp7\" (UniqueName: \"kubernetes.io/projected/3f0e5059-20d5-4772-afce-65a0c9f27c3a-kube-api-access-5xxp7\") pod \"maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp\" (UID: \"3f0e5059-20d5-4772-afce-65a0c9f27c3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp"
Sep 16 12:21:34.211679 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:34.211561    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f0e5059-20d5-4772-afce-65a0c9f27c3a-metrics-certs\") pod \"maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp\" (UID: \"3f0e5059-20d5-4772-afce-65a0c9f27c3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp"
Sep 16 12:21:34.211814 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:34.211797    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f0e5059-20d5-4772-afce-65a0c9f27c3a-maas-api-tls\") pod \"maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp\" (UID: \"3f0e5059-20d5-4772-afce-65a0c9f27c3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp"
Sep 16 12:21:34.312739 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:34.312708    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f0e5059-20d5-4772-afce-65a0c9f27c3a-openshift-service-ca\") pod \"maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp\" (UID: \"3f0e5059-20d5-4772-afce-65a0c9f27c3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp"
Sep 16 12:21:34.312739 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:34.312741    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xxp7\" (UniqueName: \"kubernetes.io/projected/3f0e5059-20d5-4772-afce-65a0c9f27c3a-kube-api-access-5xxp7\") pod \"maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp\" (UID: \"3f0e5059-20d5-4772-afce-65a0c9f27c3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp"
Sep 16 12:21:34.312966 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:34.312762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f0e5059-20d5-4772-afce-65a0c9f27c3a-metrics-certs\") pod \"maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp\" (UID: \"3f0e5059-20d5-4772-afce-65a0c9f27c3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp"
Sep 16 12:21:34.312966 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:34.312849    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f0e5059-20d5-4772-afce-65a0c9f27c3a-maas-api-tls\") pod \"maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp\" (UID: \"3f0e5059-20d5-4772-afce-65a0c9f27c3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp"
Sep 16 12:21:34.312966 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:21:34.312955    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-serving-cert: secret "maas-api-e2e-ait-1f9d3de8-serving-cert" not found
Sep 16 12:21:34.313109 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:21:34.313029    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f0e5059-20d5-4772-afce-65a0c9f27c3a-maas-api-tls podName:3f0e5059-20d5-4772-afce-65a0c9f27c3a nodeName:}" failed. No retries permitted until 2026-09-16 12:21:34.813006994 +0000 UTC m=+1088.514078300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3f0e5059-20d5-4772-afce-65a0c9f27c3a-maas-api-tls") pod "maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp" (UID: "3f0e5059-20d5-4772-afce-65a0c9f27c3a") : secret "maas-api-e2e-ait-1f9d3de8-serving-cert" not found
Sep 16 12:21:34.313432 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:34.313403    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f0e5059-20d5-4772-afce-65a0c9f27c3a-openshift-service-ca\") pod \"maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp\" (UID: \"3f0e5059-20d5-4772-afce-65a0c9f27c3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp"
Sep 16 12:21:34.315551 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:34.315525    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f0e5059-20d5-4772-afce-65a0c9f27c3a-metrics-certs\") pod \"maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp\" (UID: \"3f0e5059-20d5-4772-afce-65a0c9f27c3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp"
Sep 16 12:21:34.327598 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:34.327575    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xxp7\" (UniqueName: \"kubernetes.io/projected/3f0e5059-20d5-4772-afce-65a0c9f27c3a-kube-api-access-5xxp7\") pod \"maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp\" (UID: \"3f0e5059-20d5-4772-afce-65a0c9f27c3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp"
Sep 16 12:21:34.818158 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:34.818122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f0e5059-20d5-4772-afce-65a0c9f27c3a-maas-api-tls\") pod \"maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp\" (UID: \"3f0e5059-20d5-4772-afce-65a0c9f27c3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp"
Sep 16 12:21:34.820480 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:34.820462    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f0e5059-20d5-4772-afce-65a0c9f27c3a-maas-api-tls\") pod \"maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp\" (UID: \"3f0e5059-20d5-4772-afce-65a0c9f27c3a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp"
Sep 16 12:21:35.037061 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:35.037028    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp"
Sep 16 12:21:35.311446 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:35.311420    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp"]
Sep 16 12:21:35.312506 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:21:35.312478    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0e5059_20d5_4772_afce_65a0c9f27c3a.slice/crio-d00208776ee921171273fc79da5e0ac5a8bf53a167f711a78da32632e0b76bda WatchSource:0}: Error finding container d00208776ee921171273fc79da5e0ac5a8bf53a167f711a78da32632e0b76bda: Status 404 returned error can't find the container with id d00208776ee921171273fc79da5e0ac5a8bf53a167f711a78da32632e0b76bda
Sep 16 12:21:36.211463 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:36.211422    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp" event={"ID":"3f0e5059-20d5-4772-afce-65a0c9f27c3a","Type":"ContainerStarted","Data":"87db5d5fc4d68c717866c113e3a0101afdce692688bdc7c30476939e856d2cad"}
Sep 16 12:21:36.211648 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:36.211485    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp"
Sep 16 12:21:36.211648 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:36.211502    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp" event={"ID":"3f0e5059-20d5-4772-afce-65a0c9f27c3a","Type":"ContainerStarted","Data":"d00208776ee921171273fc79da5e0ac5a8bf53a167f711a78da32632e0b76bda"}
Sep 16 12:21:36.231752 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:36.231690    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp" podStartSLOduration=1.746671025 podStartE2EDuration="2.231670266s" totalImagesPullingTime="484.999241ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:21:36.231144268 +0000 UTC m=+1089.932215609" watchObservedRunningTime="2026-09-16 12:21:36.231670266 +0000 UTC m=+1089.932741583"
Sep 16 12:21:42.223340 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:42.223307    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp"
Sep 16 12:21:47.666255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:47.666217    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:21:48.228834 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:48.228794    2579 patch_prober.go:28] interesting pod/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 12:21:48.229052 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:48.228874    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p" podUID="6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 12:21:49.292585 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:49.292553    2579 generic.go:356] "Generic (PLEG): container finished" podID="6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" containerID="574d069cf6ea231c2135b0159a5bcfdce69c8f5ffc8e3684609ec95c8c5af066" exitCode=0
Sep 16 12:21:49.293068 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:49.292624    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p" event={"ID":"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22","Type":"ContainerDied","Data":"574d069cf6ea231c2135b0159a5bcfdce69c8f5ffc8e3684609ec95c8c5af066"}
Sep 16 12:21:49.293205 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:49.293190    2579 scope.go:117] "RemoveContainer" containerID="574d069cf6ea231c2135b0159a5bcfdce69c8f5ffc8e3684609ec95c8c5af066"
Sep 16 12:21:49.294344 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:49.294310    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:21:49.294442 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:49.294421    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:21:49.294507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:49.294492    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:21:50.305798 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:50.305751    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p" event={"ID":"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22","Type":"ContainerStarted","Data":"53bde07a6efb083347dd568c82ef93cdcb5754f9b15ea80b1ace2d0c8e827292"}
Sep 16 12:21:51.100466 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:51.100427    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:51.100884 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:51.100797    2579 patch_prober.go:28] interesting pod/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 16 12:21:51.101064 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:51.101039    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p" podUID="6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 16 12:21:52.099794 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:52.099751    2579 patch_prober.go:28] interesting pod/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 16 12:21:52.100281 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:52.099821    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p" podUID="6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 16 12:21:53.100014 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:53.099972    2579 patch_prober.go:28] interesting pod/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 16 12:21:53.100608 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:53.100574    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p" podUID="6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 16 12:21:54.099965 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:54.099919    2579 patch_prober.go:28] interesting pod/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 16 12:21:54.100185 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:54.100007    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p" podUID="6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 16 12:21:55.099918 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:55.099875    2579 patch_prober.go:28] interesting pod/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 16 12:21:55.100132 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:55.099943    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p" podUID="6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 16 12:21:56.102930 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:56.102897    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:56.103300 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:56.103176    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:21:56.103813 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:21:56.103794    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:22:03.566709 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:03.566670    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:22:04.571775 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:04.571742    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp"]
Sep 16 12:22:04.572186 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:04.571971    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp" podUID="3f0e5059-20d5-4772-afce-65a0c9f27c3a" containerName="maas-api" containerID="cri-o://87db5d5fc4d68c717866c113e3a0101afdce692688bdc7c30476939e856d2cad" gracePeriod=30
Sep 16 12:22:04.831216 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:04.831196    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp"
Sep 16 12:22:04.939310 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:04.939280    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f0e5059-20d5-4772-afce-65a0c9f27c3a-metrics-certs\") pod \"3f0e5059-20d5-4772-afce-65a0c9f27c3a\" (UID: \"3f0e5059-20d5-4772-afce-65a0c9f27c3a\") "
Sep 16 12:22:04.939492 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:04.939371    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xxp7\" (UniqueName: \"kubernetes.io/projected/3f0e5059-20d5-4772-afce-65a0c9f27c3a-kube-api-access-5xxp7\") pod \"3f0e5059-20d5-4772-afce-65a0c9f27c3a\" (UID: \"3f0e5059-20d5-4772-afce-65a0c9f27c3a\") "
Sep 16 12:22:04.939492 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:04.939450    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f0e5059-20d5-4772-afce-65a0c9f27c3a-openshift-service-ca\") pod \"3f0e5059-20d5-4772-afce-65a0c9f27c3a\" (UID: \"3f0e5059-20d5-4772-afce-65a0c9f27c3a\") "
Sep 16 12:22:04.939612 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:04.939489    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f0e5059-20d5-4772-afce-65a0c9f27c3a-maas-api-tls\") pod \"3f0e5059-20d5-4772-afce-65a0c9f27c3a\" (UID: \"3f0e5059-20d5-4772-afce-65a0c9f27c3a\") "
Sep 16 12:22:04.939900 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:04.939863    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f0e5059-20d5-4772-afce-65a0c9f27c3a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3f0e5059-20d5-4772-afce-65a0c9f27c3a" (UID: "3f0e5059-20d5-4772-afce-65a0c9f27c3a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:22:04.941591 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:04.941568    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f0e5059-20d5-4772-afce-65a0c9f27c3a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3f0e5059-20d5-4772-afce-65a0c9f27c3a" (UID: "3f0e5059-20d5-4772-afce-65a0c9f27c3a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:22:04.941869 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:04.941843    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f0e5059-20d5-4772-afce-65a0c9f27c3a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3f0e5059-20d5-4772-afce-65a0c9f27c3a" (UID: "3f0e5059-20d5-4772-afce-65a0c9f27c3a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:22:04.941932 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:04.941850    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f0e5059-20d5-4772-afce-65a0c9f27c3a-kube-api-access-5xxp7" (OuterVolumeSpecName: "kube-api-access-5xxp7") pod "3f0e5059-20d5-4772-afce-65a0c9f27c3a" (UID: "3f0e5059-20d5-4772-afce-65a0c9f27c3a"). InnerVolumeSpecName "kube-api-access-5xxp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:22:05.041098 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:05.041066    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f0e5059-20d5-4772-afce-65a0c9f27c3a-metrics-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:05.041098 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:05.041095    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xxp7\" (UniqueName: \"kubernetes.io/projected/3f0e5059-20d5-4772-afce-65a0c9f27c3a-kube-api-access-5xxp7\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:05.041310 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:05.041107    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f0e5059-20d5-4772-afce-65a0c9f27c3a-openshift-service-ca\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:05.041310 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:05.041119    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f0e5059-20d5-4772-afce-65a0c9f27c3a-maas-api-tls\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:05.404955 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:05.404915    2579 generic.go:356] "Generic (PLEG): container finished" podID="3f0e5059-20d5-4772-afce-65a0c9f27c3a" containerID="87db5d5fc4d68c717866c113e3a0101afdce692688bdc7c30476939e856d2cad" exitCode=0
Sep 16 12:22:05.405140 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:05.404960    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp"
Sep 16 12:22:05.405140 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:05.405002    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp" event={"ID":"3f0e5059-20d5-4772-afce-65a0c9f27c3a","Type":"ContainerDied","Data":"87db5d5fc4d68c717866c113e3a0101afdce692688bdc7c30476939e856d2cad"}
Sep 16 12:22:05.405140 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:05.405061    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp" event={"ID":"3f0e5059-20d5-4772-afce-65a0c9f27c3a","Type":"ContainerDied","Data":"d00208776ee921171273fc79da5e0ac5a8bf53a167f711a78da32632e0b76bda"}
Sep 16 12:22:05.405140 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:05.405087    2579 scope.go:117] "RemoveContainer" containerID="87db5d5fc4d68c717866c113e3a0101afdce692688bdc7c30476939e856d2cad"
Sep 16 12:22:05.415073 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:05.415053    2579 scope.go:117] "RemoveContainer" containerID="87db5d5fc4d68c717866c113e3a0101afdce692688bdc7c30476939e856d2cad"
Sep 16 12:22:05.415340 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:22:05.415322    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87db5d5fc4d68c717866c113e3a0101afdce692688bdc7c30476939e856d2cad\": container with ID starting with 87db5d5fc4d68c717866c113e3a0101afdce692688bdc7c30476939e856d2cad not found: ID does not exist" containerID="87db5d5fc4d68c717866c113e3a0101afdce692688bdc7c30476939e856d2cad"
Sep 16 12:22:05.415403 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:05.415346    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87db5d5fc4d68c717866c113e3a0101afdce692688bdc7c30476939e856d2cad"} err="failed to get container status \"87db5d5fc4d68c717866c113e3a0101afdce692688bdc7c30476939e856d2cad\": rpc error: code = NotFound desc = could not find container \"87db5d5fc4d68c717866c113e3a0101afdce692688bdc7c30476939e856d2cad\": container with ID starting with 87db5d5fc4d68c717866c113e3a0101afdce692688bdc7c30476939e856d2cad not found: ID does not exist"
Sep 16 12:22:05.427852 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:05.427831    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp"]
Sep 16 12:22:05.433209 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:05.433186    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1f9d3de8-5d76498bb7-d7fkp"]
Sep 16 12:22:05.944894 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:05.944861    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"]
Sep 16 12:22:05.945364 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:05.945113    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p" podUID="6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" containerName="istio-proxy" containerID="cri-o://53bde07a6efb083347dd568c82ef93cdcb5754f9b15ea80b1ace2d0c8e827292" gracePeriod=30
Sep 16 12:22:06.664421 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.664385    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"]
Sep 16 12:22:06.664844 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.664829    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f0e5059-20d5-4772-afce-65a0c9f27c3a" containerName="maas-api"
Sep 16 12:22:06.664906 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.664846    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f0e5059-20d5-4772-afce-65a0c9f27c3a" containerName="maas-api"
Sep 16 12:22:06.664942 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.664933    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f0e5059-20d5-4772-afce-65a0c9f27c3a" containerName="maas-api"
Sep 16 12:22:06.668058 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.668041    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.670495 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.670471    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-c2aae110-openshift-default-dockercfg-ch9rs\""
Sep 16 12:22:06.681598 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.681574    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"]
Sep 16 12:22:06.758148 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.757584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-istio-envoy\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.758148 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.757644    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms7rk\" (UniqueName: \"kubernetes.io/projected/2f17ee56-1a01-4732-8359-4dea9620fc8b-kube-api-access-ms7rk\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.758148 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.757705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-istio-data\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.758148 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.757732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-workload-certs\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.758148 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.757774    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f17ee56-1a01-4732-8359-4dea9620fc8b-istio-token\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.758148 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.757826    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f17ee56-1a01-4732-8359-4dea9620fc8b-istiod-ca-cert\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.758148 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.757850    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-workload-socket\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.758148 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.757919    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-credential-socket\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.758148 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.757944    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f17ee56-1a01-4732-8359-4dea9620fc8b-istio-podinfo\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.793394 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.793351    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f0e5059-20d5-4772-afce-65a0c9f27c3a" path="/var/lib/kubelet/pods/3f0e5059-20d5-4772-afce-65a0c9f27c3a/volumes"
Sep 16 12:22:06.858831 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.858793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms7rk\" (UniqueName: \"kubernetes.io/projected/2f17ee56-1a01-4732-8359-4dea9620fc8b-kube-api-access-ms7rk\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.859027 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.858844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-istio-data\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.859027 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.858865    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-workload-certs\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.859027 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.858895    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f17ee56-1a01-4732-8359-4dea9620fc8b-istio-token\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.859027 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.858929    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f17ee56-1a01-4732-8359-4dea9620fc8b-istiod-ca-cert\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.859027 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.858945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-workload-socket\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.859027 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.858986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-credential-socket\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.859027 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.859007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f17ee56-1a01-4732-8359-4dea9620fc8b-istio-podinfo\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.859419 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.859056    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-istio-envoy\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.859419 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.859268    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-istio-data\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.859419 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.859294    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-workload-certs\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.859607 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.859583    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-credential-socket\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.859699 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.859591    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-workload-socket\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.859844 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.859819    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f17ee56-1a01-4732-8359-4dea9620fc8b-istiod-ca-cert\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.861360 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.861340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-istio-envoy\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.861630 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.861610    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f17ee56-1a01-4732-8359-4dea9620fc8b-istio-podinfo\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.867269 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.867250    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f17ee56-1a01-4732-8359-4dea9620fc8b-istio-token\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.867339 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.867320    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms7rk\" (UniqueName: \"kubernetes.io/projected/2f17ee56-1a01-4732-8359-4dea9620fc8b-kube-api-access-ms7rk\") pod \"e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") " pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:06.978855 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:06.978765    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:07.260860 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:07.260829    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"]
Sep 16 12:22:07.262296 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:22:07.262270    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f17ee56_1a01_4732_8359_4dea9620fc8b.slice/crio-d80fd23b02e46f4a1bb7b996d6bd493cc4e44ea9872b29a7b094b79a03522e4d WatchSource:0}: Error finding container d80fd23b02e46f4a1bb7b996d6bd493cc4e44ea9872b29a7b094b79a03522e4d: Status 404 returned error can't find the container with id d80fd23b02e46f4a1bb7b996d6bd493cc4e44ea9872b29a7b094b79a03522e4d
Sep 16 12:22:07.264766 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:07.264733    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:22:07.264853 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:07.264797    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:22:07.264853 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:07.264825    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:22:07.419732 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:07.419694    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw" event={"ID":"2f17ee56-1a01-4732-8359-4dea9620fc8b","Type":"ContainerStarted","Data":"07290b5772f7eecfe766e21ee9ad8875d87f108e3c87ff96ef26db0c36b6ad30"}
Sep 16 12:22:07.419936 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:07.419754    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw" event={"ID":"2f17ee56-1a01-4732-8359-4dea9620fc8b","Type":"ContainerStarted","Data":"d80fd23b02e46f4a1bb7b996d6bd493cc4e44ea9872b29a7b094b79a03522e4d"}
Sep 16 12:22:07.438443 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:07.438386    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw" podStartSLOduration=1.4383687410000001 podStartE2EDuration="1.438368741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:22:07.437077709 +0000 UTC m=+1121.138149037" watchObservedRunningTime="2026-09-16 12:22:07.438368741 +0000 UTC m=+1121.139440066"
Sep 16 12:22:07.978844 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:07.978812    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:08.984367 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:08.984337    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:09.434634 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:09.434601    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:09.438365 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:09.438340    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:11.105670 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.105623    2579 patch_prober.go:28] interesting pod/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Sep 16 12:22:11.106151 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.105703    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p" podUID="6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Sep 16 12:22:11.206363 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.206299    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:22:11.306069 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.306027    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hq9j7\" (UniqueName: \"kubernetes.io/projected/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-kube-api-access-hq9j7\") pod \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") "
Sep 16 12:22:11.306265 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.306092    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-workload-socket\") pod \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") "
Sep 16 12:22:11.306265 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.306119    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istiod-ca-cert\") pod \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") "
Sep 16 12:22:11.306265 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.306159    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istio-data\") pod \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") "
Sep 16 12:22:11.306265 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.306228    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istio-envoy\") pod \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") "
Sep 16 12:22:11.306265 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.306248    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-credential-socket\") pod \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") "
Sep 16 12:22:11.306539 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.306293    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istio-podinfo\") pod \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") "
Sep 16 12:22:11.306539 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.306326    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-workload-certs\") pod \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") "
Sep 16 12:22:11.306539 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.306360    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istio-token\") pod \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\" (UID: \"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22\") "
Sep 16 12:22:11.306539 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.306420    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" (UID: "6d8af3a6-2d34-4b3f-ab9f-676c8450ab22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:22:11.306539 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.306512    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" (UID: "6d8af3a6-2d34-4b3f-ab9f-676c8450ab22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:22:11.306837 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.306535    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istio-data" (OuterVolumeSpecName: "istio-data") pod "6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" (UID: "6d8af3a6-2d34-4b3f-ab9f-676c8450ab22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:22:11.306837 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.306550    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" (UID: "6d8af3a6-2d34-4b3f-ab9f-676c8450ab22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:22:11.306837 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.306739    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" (UID: "6d8af3a6-2d34-4b3f-ab9f-676c8450ab22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:22:11.306981 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.306860    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-credential-socket\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:11.306981 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.306881    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-workload-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:11.306981 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.306895    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-workload-socket\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:11.306981 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.306907    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istiod-ca-cert\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:11.306981 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.306920    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istio-data\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:11.308401 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.308373    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-kube-api-access-hq9j7" (OuterVolumeSpecName: "kube-api-access-hq9j7") pod "6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" (UID: "6d8af3a6-2d34-4b3f-ab9f-676c8450ab22"). InnerVolumeSpecName "kube-api-access-hq9j7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:22:11.308401 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.308388    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" (UID: "6d8af3a6-2d34-4b3f-ab9f-676c8450ab22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:22:11.308870 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.308853    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istio-token" (OuterVolumeSpecName: "istio-token") pod "6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" (UID: "6d8af3a6-2d34-4b3f-ab9f-676c8450ab22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:22:11.308951 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.308934    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" (UID: "6d8af3a6-2d34-4b3f-ab9f-676c8450ab22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:22:11.408019 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.407977    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istio-envoy\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:11.408019 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.408010    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istio-podinfo\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:11.408019 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.408023    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-istio-token\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:11.408266 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.408035    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq9j7\" (UniqueName: \"kubernetes.io/projected/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22-kube-api-access-hq9j7\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:11.447064 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.447032    2579 generic.go:356] "Generic (PLEG): container finished" podID="6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" containerID="53bde07a6efb083347dd568c82ef93cdcb5754f9b15ea80b1ace2d0c8e827292" exitCode=0
Sep 16 12:22:11.447248 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.447087    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"
Sep 16 12:22:11.447248 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.447115    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p" event={"ID":"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22","Type":"ContainerDied","Data":"53bde07a6efb083347dd568c82ef93cdcb5754f9b15ea80b1ace2d0c8e827292"}
Sep 16 12:22:11.447248 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.447175    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p" event={"ID":"6d8af3a6-2d34-4b3f-ab9f-676c8450ab22","Type":"ContainerDied","Data":"1d40491f7e1ffc8ffd331896a55665b244106c5d168ec717861b34862d7b1ab3"}
Sep 16 12:22:11.447248 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.447206    2579 scope.go:117] "RemoveContainer" containerID="53bde07a6efb083347dd568c82ef93cdcb5754f9b15ea80b1ace2d0c8e827292"
Sep 16 12:22:11.457277 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.457256    2579 scope.go:117] "RemoveContainer" containerID="574d069cf6ea231c2135b0159a5bcfdce69c8f5ffc8e3684609ec95c8c5af066"
Sep 16 12:22:11.466424 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.466401    2579 scope.go:117] "RemoveContainer" containerID="53bde07a6efb083347dd568c82ef93cdcb5754f9b15ea80b1ace2d0c8e827292"
Sep 16 12:22:11.467324 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:22:11.467104    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53bde07a6efb083347dd568c82ef93cdcb5754f9b15ea80b1ace2d0c8e827292\": container with ID starting with 53bde07a6efb083347dd568c82ef93cdcb5754f9b15ea80b1ace2d0c8e827292 not found: ID does not exist" containerID="53bde07a6efb083347dd568c82ef93cdcb5754f9b15ea80b1ace2d0c8e827292"
Sep 16 12:22:11.467324 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.467144    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53bde07a6efb083347dd568c82ef93cdcb5754f9b15ea80b1ace2d0c8e827292"} err="failed to get container status \"53bde07a6efb083347dd568c82ef93cdcb5754f9b15ea80b1ace2d0c8e827292\": rpc error: code = NotFound desc = could not find container \"53bde07a6efb083347dd568c82ef93cdcb5754f9b15ea80b1ace2d0c8e827292\": container with ID starting with 53bde07a6efb083347dd568c82ef93cdcb5754f9b15ea80b1ace2d0c8e827292 not found: ID does not exist"
Sep 16 12:22:11.467324 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.467169    2579 scope.go:117] "RemoveContainer" containerID="574d069cf6ea231c2135b0159a5bcfdce69c8f5ffc8e3684609ec95c8c5af066"
Sep 16 12:22:11.467535 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:22:11.467514    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"574d069cf6ea231c2135b0159a5bcfdce69c8f5ffc8e3684609ec95c8c5af066\": container with ID starting with 574d069cf6ea231c2135b0159a5bcfdce69c8f5ffc8e3684609ec95c8c5af066 not found: ID does not exist" containerID="574d069cf6ea231c2135b0159a5bcfdce69c8f5ffc8e3684609ec95c8c5af066"
Sep 16 12:22:11.467583 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.467543    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574d069cf6ea231c2135b0159a5bcfdce69c8f5ffc8e3684609ec95c8c5af066"} err="failed to get container status \"574d069cf6ea231c2135b0159a5bcfdce69c8f5ffc8e3684609ec95c8c5af066\": rpc error: code = NotFound desc = could not find container \"574d069cf6ea231c2135b0159a5bcfdce69c8f5ffc8e3684609ec95c8c5af066\": container with ID starting with 574d069cf6ea231c2135b0159a5bcfdce69c8f5ffc8e3684609ec95c8c5af066 not found: ID does not exist"
Sep 16 12:22:11.470177 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.470156    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"]
Sep 16 12:22:11.473783 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:11.473761    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1f9d3de8-openshift-default-558d9746fb-g427p"]
Sep 16 12:22:12.649058 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:12.649025    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp"]
Sep 16 12:22:12.649535 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:12.649435    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" containerName="istio-proxy"
Sep 16 12:22:12.649535 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:12.649447    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" containerName="istio-proxy"
Sep 16 12:22:12.649535 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:12.649524    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" containerName="istio-proxy"
Sep 16 12:22:12.652387 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:12.652368    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp"
Sep 16 12:22:12.654986 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:12.654939    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:22:12.654986 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:12.654952    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-c2aae110-serving-cert\""
Sep 16 12:22:12.655174 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:12.655111    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 12:22:12.656296 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:12.656273    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4gkmq\""
Sep 16 12:22:12.656413 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:12.656321    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 12:22:12.662984 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:12.662958    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp"]
Sep 16 12:22:12.722461 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:12.722419    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f0d8036-7963-408b-9782-73ff346922a3-maas-api-tls\") pod \"maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp\" (UID: \"9f0d8036-7963-408b-9782-73ff346922a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp"
Sep 16 12:22:12.722646 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:12.722474    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f0d8036-7963-408b-9782-73ff346922a3-openshift-service-ca\") pod \"maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp\" (UID: \"9f0d8036-7963-408b-9782-73ff346922a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp"
Sep 16 12:22:12.722646 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:12.722547    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzd8l\" (UniqueName: \"kubernetes.io/projected/9f0d8036-7963-408b-9782-73ff346922a3-kube-api-access-fzd8l\") pod \"maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp\" (UID: \"9f0d8036-7963-408b-9782-73ff346922a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp"
Sep 16 12:22:12.722646 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:12.722570    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f0d8036-7963-408b-9782-73ff346922a3-metrics-certs\") pod \"maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp\" (UID: \"9f0d8036-7963-408b-9782-73ff346922a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp"
Sep 16 12:22:12.781807 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:12.781770    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" path="/var/lib/kubelet/pods/6d8af3a6-2d34-4b3f-ab9f-676c8450ab22/volumes"
Sep 16 12:22:12.823076 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:12.823042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzd8l\" (UniqueName: \"kubernetes.io/projected/9f0d8036-7963-408b-9782-73ff346922a3-kube-api-access-fzd8l\") pod \"maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp\" (UID: \"9f0d8036-7963-408b-9782-73ff346922a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp"
Sep 16 12:22:12.823076 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:12.823082    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f0d8036-7963-408b-9782-73ff346922a3-metrics-certs\") pod \"maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp\" (UID: \"9f0d8036-7963-408b-9782-73ff346922a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp"
Sep 16 12:22:12.823346 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:12.823173    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f0d8036-7963-408b-9782-73ff346922a3-maas-api-tls\") pod \"maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp\" (UID: \"9f0d8036-7963-408b-9782-73ff346922a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp"
Sep 16 12:22:12.823346 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:12.823236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f0d8036-7963-408b-9782-73ff346922a3-openshift-service-ca\") pod \"maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp\" (UID: \"9f0d8036-7963-408b-9782-73ff346922a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp"
Sep 16 12:22:12.823346 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:22:12.823285    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-serving-cert: secret "maas-api-e2e-derive-c2aae110-serving-cert" not found
Sep 16 12:22:12.823514 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:22:12.823358    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f0d8036-7963-408b-9782-73ff346922a3-maas-api-tls podName:9f0d8036-7963-408b-9782-73ff346922a3 nodeName:}" failed. No retries permitted until 2026-09-16 12:22:13.323336566 +0000 UTC m=+1127.024407872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9f0d8036-7963-408b-9782-73ff346922a3-maas-api-tls") pod "maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp" (UID: "9f0d8036-7963-408b-9782-73ff346922a3") : secret "maas-api-e2e-derive-c2aae110-serving-cert" not found
Sep 16 12:22:12.823974 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:12.823953    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f0d8036-7963-408b-9782-73ff346922a3-openshift-service-ca\") pod \"maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp\" (UID: \"9f0d8036-7963-408b-9782-73ff346922a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp"
Sep 16 12:22:12.825545 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:12.825522    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f0d8036-7963-408b-9782-73ff346922a3-metrics-certs\") pod \"maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp\" (UID: \"9f0d8036-7963-408b-9782-73ff346922a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp"
Sep 16 12:22:12.833550 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:12.833518    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzd8l\" (UniqueName: \"kubernetes.io/projected/9f0d8036-7963-408b-9782-73ff346922a3-kube-api-access-fzd8l\") pod \"maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp\" (UID: \"9f0d8036-7963-408b-9782-73ff346922a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp"
Sep 16 12:22:13.328576 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:13.328541    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f0d8036-7963-408b-9782-73ff346922a3-maas-api-tls\") pod \"maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp\" (UID: \"9f0d8036-7963-408b-9782-73ff346922a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp"
Sep 16 12:22:13.331257 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:13.331224    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f0d8036-7963-408b-9782-73ff346922a3-maas-api-tls\") pod \"maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp\" (UID: \"9f0d8036-7963-408b-9782-73ff346922a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp"
Sep 16 12:22:13.564974 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:13.564935    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp"
Sep 16 12:22:13.836685 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:13.836633    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp"]
Sep 16 12:22:13.838459 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:22:13.838435    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0d8036_7963_408b_9782_73ff346922a3.slice/crio-04696b8adf71732ad69706c45ac2d6bab2eec1939689f53fcdce8c428d64a761 WatchSource:0}: Error finding container 04696b8adf71732ad69706c45ac2d6bab2eec1939689f53fcdce8c428d64a761: Status 404 returned error can't find the container with id 04696b8adf71732ad69706c45ac2d6bab2eec1939689f53fcdce8c428d64a761
Sep 16 12:22:14.467312 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:14.467272    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp" event={"ID":"9f0d8036-7963-408b-9782-73ff346922a3","Type":"ContainerStarted","Data":"054c4f90eeae8cc32c6caf1fe6de59d3f4758f713e0d3a7573089f9bc993ba6d"}
Sep 16 12:22:14.467480 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:14.467331    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp" event={"ID":"9f0d8036-7963-408b-9782-73ff346922a3","Type":"ContainerStarted","Data":"04696b8adf71732ad69706c45ac2d6bab2eec1939689f53fcdce8c428d64a761"}
Sep 16 12:22:14.467480 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:14.467357    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp"
Sep 16 12:22:14.483768 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:14.483710    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp" podStartSLOduration=2.057572379 podStartE2EDuration="2.483689104s" totalImagesPullingTime="426.116725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:22:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:22:14.482639106 +0000 UTC m=+1128.183710433" watchObservedRunningTime="2026-09-16 12:22:14.483689104 +0000 UTC m=+1128.184760430"
Sep 16 12:22:20.479295 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:20.479261    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp"
Sep 16 12:22:25.171403 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:25.171364    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:22:35.608778 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:35.608741    2579 generic.go:356] "Generic (PLEG): container finished" podID="2f17ee56-1a01-4732-8359-4dea9620fc8b" containerID="07290b5772f7eecfe766e21ee9ad8875d87f108e3c87ff96ef26db0c36b6ad30" exitCode=0
Sep 16 12:22:35.609176 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:35.608814    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw" event={"ID":"2f17ee56-1a01-4732-8359-4dea9620fc8b","Type":"ContainerDied","Data":"07290b5772f7eecfe766e21ee9ad8875d87f108e3c87ff96ef26db0c36b6ad30"}
Sep 16 12:22:35.609258 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:35.609244    2579 scope.go:117] "RemoveContainer" containerID="07290b5772f7eecfe766e21ee9ad8875d87f108e3c87ff96ef26db0c36b6ad30"
Sep 16 12:22:35.610394 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:35.610363    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:22:35.610512 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:35.610442    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:22:35.610512 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:35.610487    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:22:36.616729 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:36.616692    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw" event={"ID":"2f17ee56-1a01-4732-8359-4dea9620fc8b","Type":"ContainerStarted","Data":"bff8a1887121de736637b73f047dcb2abaa5854e313cfb99697ae4a163ea4f23"}
Sep 16 12:22:36.979338 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:36.978942    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:36.981769 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:36.981640    2579 patch_prober.go:28] interesting pod/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 16 12:22:36.982085 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:36.981969    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw" podUID="2f17ee56-1a01-4732-8359-4dea9620fc8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 16 12:22:37.979336 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:37.979301    2579 patch_prober.go:28] interesting pod/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 16 12:22:37.979956 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:37.979922    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw" podUID="2f17ee56-1a01-4732-8359-4dea9620fc8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 16 12:22:38.979990 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:38.979939    2579 patch_prober.go:28] interesting pod/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 16 12:22:38.980499 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:38.980013    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw" podUID="2f17ee56-1a01-4732-8359-4dea9620fc8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 16 12:22:39.202679 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.201730    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp"]
Sep 16 12:22:39.202679 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.201980    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp" podUID="9f0d8036-7963-408b-9782-73ff346922a3" containerName="maas-api" containerID="cri-o://054c4f90eeae8cc32c6caf1fe6de59d3f4758f713e0d3a7573089f9bc993ba6d" gracePeriod=30
Sep 16 12:22:39.435243 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.435209    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:39.468200 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.468176    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp"
Sep 16 12:22:39.515958 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.515877    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzd8l\" (UniqueName: \"kubernetes.io/projected/9f0d8036-7963-408b-9782-73ff346922a3-kube-api-access-fzd8l\") pod \"9f0d8036-7963-408b-9782-73ff346922a3\" (UID: \"9f0d8036-7963-408b-9782-73ff346922a3\") "
Sep 16 12:22:39.515958 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.515929    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f0d8036-7963-408b-9782-73ff346922a3-maas-api-tls\") pod \"9f0d8036-7963-408b-9782-73ff346922a3\" (UID: \"9f0d8036-7963-408b-9782-73ff346922a3\") "
Sep 16 12:22:39.516141 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.515985    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f0d8036-7963-408b-9782-73ff346922a3-openshift-service-ca\") pod \"9f0d8036-7963-408b-9782-73ff346922a3\" (UID: \"9f0d8036-7963-408b-9782-73ff346922a3\") "
Sep 16 12:22:39.516141 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.516021    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f0d8036-7963-408b-9782-73ff346922a3-metrics-certs\") pod \"9f0d8036-7963-408b-9782-73ff346922a3\" (UID: \"9f0d8036-7963-408b-9782-73ff346922a3\") "
Sep 16 12:22:39.516491 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.516468    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0d8036-7963-408b-9782-73ff346922a3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9f0d8036-7963-408b-9782-73ff346922a3" (UID: "9f0d8036-7963-408b-9782-73ff346922a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:22:39.518629 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.518596    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0d8036-7963-408b-9782-73ff346922a3-kube-api-access-fzd8l" (OuterVolumeSpecName: "kube-api-access-fzd8l") pod "9f0d8036-7963-408b-9782-73ff346922a3" (UID: "9f0d8036-7963-408b-9782-73ff346922a3"). InnerVolumeSpecName "kube-api-access-fzd8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:22:39.518805 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.518623    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f0d8036-7963-408b-9782-73ff346922a3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9f0d8036-7963-408b-9782-73ff346922a3" (UID: "9f0d8036-7963-408b-9782-73ff346922a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:22:39.518805 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.518645    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f0d8036-7963-408b-9782-73ff346922a3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9f0d8036-7963-408b-9782-73ff346922a3" (UID: "9f0d8036-7963-408b-9782-73ff346922a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:22:39.614285 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.614239    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"]
Sep 16 12:22:39.617477 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.617387    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f0d8036-7963-408b-9782-73ff346922a3-maas-api-tls\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:39.617477 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.617423    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f0d8036-7963-408b-9782-73ff346922a3-openshift-service-ca\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:39.617477 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.617439    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f0d8036-7963-408b-9782-73ff346922a3-metrics-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:39.617477 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.617453    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzd8l\" (UniqueName: \"kubernetes.io/projected/9f0d8036-7963-408b-9782-73ff346922a3-kube-api-access-fzd8l\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:39.641065 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.640981    2579 generic.go:356] "Generic (PLEG): container finished" podID="9f0d8036-7963-408b-9782-73ff346922a3" containerID="054c4f90eeae8cc32c6caf1fe6de59d3f4758f713e0d3a7573089f9bc993ba6d" exitCode=0
Sep 16 12:22:39.641065 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.641044    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp"
Sep 16 12:22:39.641065 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.641060    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp" event={"ID":"9f0d8036-7963-408b-9782-73ff346922a3","Type":"ContainerDied","Data":"054c4f90eeae8cc32c6caf1fe6de59d3f4758f713e0d3a7573089f9bc993ba6d"}
Sep 16 12:22:39.641298 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.641115    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp" event={"ID":"9f0d8036-7963-408b-9782-73ff346922a3","Type":"ContainerDied","Data":"04696b8adf71732ad69706c45ac2d6bab2eec1939689f53fcdce8c428d64a761"}
Sep 16 12:22:39.641298 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.641137    2579 scope.go:117] "RemoveContainer" containerID="054c4f90eeae8cc32c6caf1fe6de59d3f4758f713e0d3a7573089f9bc993ba6d"
Sep 16 12:22:39.641435 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.641415    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw" podUID="2f17ee56-1a01-4732-8359-4dea9620fc8b" containerName="istio-proxy" containerID="cri-o://bff8a1887121de736637b73f047dcb2abaa5854e313cfb99697ae4a163ea4f23" gracePeriod=30
Sep 16 12:22:39.651101 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.651082    2579 scope.go:117] "RemoveContainer" containerID="054c4f90eeae8cc32c6caf1fe6de59d3f4758f713e0d3a7573089f9bc993ba6d"
Sep 16 12:22:39.651335 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:22:39.651316    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"054c4f90eeae8cc32c6caf1fe6de59d3f4758f713e0d3a7573089f9bc993ba6d\": container with ID starting with 054c4f90eeae8cc32c6caf1fe6de59d3f4758f713e0d3a7573089f9bc993ba6d not found: ID does not exist" containerID="054c4f90eeae8cc32c6caf1fe6de59d3f4758f713e0d3a7573089f9bc993ba6d"
Sep 16 12:22:39.651411 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.651345    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"054c4f90eeae8cc32c6caf1fe6de59d3f4758f713e0d3a7573089f9bc993ba6d"} err="failed to get container status \"054c4f90eeae8cc32c6caf1fe6de59d3f4758f713e0d3a7573089f9bc993ba6d\": rpc error: code = NotFound desc = could not find container \"054c4f90eeae8cc32c6caf1fe6de59d3f4758f713e0d3a7573089f9bc993ba6d\": container with ID starting with 054c4f90eeae8cc32c6caf1fe6de59d3f4758f713e0d3a7573089f9bc993ba6d not found: ID does not exist"
Sep 16 12:22:39.717684 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.715724    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp"]
Sep 16 12:22:39.717684 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:39.716136    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c2aae110-7f488986d6-d8mmp"]
Sep 16 12:22:40.784350 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:40.784307    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f0d8036-7963-408b-9782-73ff346922a3" path="/var/lib/kubelet/pods/9f0d8036-7963-408b-9782-73ff346922a3/volumes"
Sep 16 12:22:42.062424 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:42.062387    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:22:46.987490 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:46.987461    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:47.094064 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.094033    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-credential-socket\") pod \"2f17ee56-1a01-4732-8359-4dea9620fc8b\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") "
Sep 16 12:22:47.094241 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.094090    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms7rk\" (UniqueName: \"kubernetes.io/projected/2f17ee56-1a01-4732-8359-4dea9620fc8b-kube-api-access-ms7rk\") pod \"2f17ee56-1a01-4732-8359-4dea9620fc8b\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") "
Sep 16 12:22:47.094241 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.094117    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f17ee56-1a01-4732-8359-4dea9620fc8b-istio-token\") pod \"2f17ee56-1a01-4732-8359-4dea9620fc8b\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") "
Sep 16 12:22:47.094241 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.094174    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-workload-socket\") pod \"2f17ee56-1a01-4732-8359-4dea9620fc8b\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") "
Sep 16 12:22:47.094363 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.094334    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-istio-envoy\") pod \"2f17ee56-1a01-4732-8359-4dea9620fc8b\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") "
Sep 16 12:22:47.094415 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.094383    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-istio-data\") pod \"2f17ee56-1a01-4732-8359-4dea9620fc8b\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") "
Sep 16 12:22:47.094415 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.094331    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2f17ee56-1a01-4732-8359-4dea9620fc8b" (UID: "2f17ee56-1a01-4732-8359-4dea9620fc8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:22:47.094415 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.094408    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-workload-certs\") pod \"2f17ee56-1a01-4732-8359-4dea9620fc8b\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") "
Sep 16 12:22:47.094552 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.094438    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2f17ee56-1a01-4732-8359-4dea9620fc8b" (UID: "2f17ee56-1a01-4732-8359-4dea9620fc8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:22:47.094552 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.094443    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f17ee56-1a01-4732-8359-4dea9620fc8b-istiod-ca-cert\") pod \"2f17ee56-1a01-4732-8359-4dea9620fc8b\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") "
Sep 16 12:22:47.094552 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.094521    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f17ee56-1a01-4732-8359-4dea9620fc8b-istio-podinfo\") pod \"2f17ee56-1a01-4732-8359-4dea9620fc8b\" (UID: \"2f17ee56-1a01-4732-8359-4dea9620fc8b\") "
Sep 16 12:22:47.094743 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.094630    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-istio-data" (OuterVolumeSpecName: "istio-data") pod "2f17ee56-1a01-4732-8359-4dea9620fc8b" (UID: "2f17ee56-1a01-4732-8359-4dea9620fc8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:22:47.094800 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.094743    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2f17ee56-1a01-4732-8359-4dea9620fc8b" (UID: "2f17ee56-1a01-4732-8359-4dea9620fc8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:22:47.094800 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.094748    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f17ee56-1a01-4732-8359-4dea9620fc8b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2f17ee56-1a01-4732-8359-4dea9620fc8b" (UID: "2f17ee56-1a01-4732-8359-4dea9620fc8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:22:47.094940 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.094924    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-workload-socket\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:47.094995 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.094945    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-istio-data\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:47.094995 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.094959    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-workload-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:47.094995 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.094970    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f17ee56-1a01-4732-8359-4dea9620fc8b-istiod-ca-cert\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:47.094995 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.094977    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-credential-socket\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:47.096459 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.096425    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f17ee56-1a01-4732-8359-4dea9620fc8b-kube-api-access-ms7rk" (OuterVolumeSpecName: "kube-api-access-ms7rk") pod "2f17ee56-1a01-4732-8359-4dea9620fc8b" (UID: "2f17ee56-1a01-4732-8359-4dea9620fc8b"). InnerVolumeSpecName "kube-api-access-ms7rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:22:47.096731 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.096708    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f17ee56-1a01-4732-8359-4dea9620fc8b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2f17ee56-1a01-4732-8359-4dea9620fc8b" (UID: "2f17ee56-1a01-4732-8359-4dea9620fc8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:22:47.096799 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.096775    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2f17ee56-1a01-4732-8359-4dea9620fc8b" (UID: "2f17ee56-1a01-4732-8359-4dea9620fc8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:22:47.096863 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.096796    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f17ee56-1a01-4732-8359-4dea9620fc8b-istio-token" (OuterVolumeSpecName: "istio-token") pod "2f17ee56-1a01-4732-8359-4dea9620fc8b" (UID: "2f17ee56-1a01-4732-8359-4dea9620fc8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:22:47.196336 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.196302    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f17ee56-1a01-4732-8359-4dea9620fc8b-istio-podinfo\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:47.196336 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.196332    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms7rk\" (UniqueName: \"kubernetes.io/projected/2f17ee56-1a01-4732-8359-4dea9620fc8b-kube-api-access-ms7rk\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:47.196336 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.196342    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f17ee56-1a01-4732-8359-4dea9620fc8b-istio-token\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:47.196563 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.196350    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f17ee56-1a01-4732-8359-4dea9620fc8b-istio-envoy\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:22:47.692449 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.692417    2579 generic.go:356] "Generic (PLEG): container finished" podID="2f17ee56-1a01-4732-8359-4dea9620fc8b" containerID="bff8a1887121de736637b73f047dcb2abaa5854e313cfb99697ae4a163ea4f23" exitCode=0
Sep 16 12:22:47.692628 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.692464    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"
Sep 16 12:22:47.692628 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.692507    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw" event={"ID":"2f17ee56-1a01-4732-8359-4dea9620fc8b","Type":"ContainerDied","Data":"bff8a1887121de736637b73f047dcb2abaa5854e313cfb99697ae4a163ea4f23"}
Sep 16 12:22:47.692628 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.692562    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw" event={"ID":"2f17ee56-1a01-4732-8359-4dea9620fc8b","Type":"ContainerDied","Data":"d80fd23b02e46f4a1bb7b996d6bd493cc4e44ea9872b29a7b094b79a03522e4d"}
Sep 16 12:22:47.692628 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.692584    2579 scope.go:117] "RemoveContainer" containerID="bff8a1887121de736637b73f047dcb2abaa5854e313cfb99697ae4a163ea4f23"
Sep 16 12:22:47.702966 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.702949    2579 scope.go:117] "RemoveContainer" containerID="07290b5772f7eecfe766e21ee9ad8875d87f108e3c87ff96ef26db0c36b6ad30"
Sep 16 12:22:47.711233 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.711194    2579 scope.go:117] "RemoveContainer" containerID="bff8a1887121de736637b73f047dcb2abaa5854e313cfb99697ae4a163ea4f23"
Sep 16 12:22:47.711618 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:22:47.711588    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bff8a1887121de736637b73f047dcb2abaa5854e313cfb99697ae4a163ea4f23\": container with ID starting with bff8a1887121de736637b73f047dcb2abaa5854e313cfb99697ae4a163ea4f23 not found: ID does not exist" containerID="bff8a1887121de736637b73f047dcb2abaa5854e313cfb99697ae4a163ea4f23"
Sep 16 12:22:47.711739 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.711626    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bff8a1887121de736637b73f047dcb2abaa5854e313cfb99697ae4a163ea4f23"} err="failed to get container status \"bff8a1887121de736637b73f047dcb2abaa5854e313cfb99697ae4a163ea4f23\": rpc error: code = NotFound desc = could not find container \"bff8a1887121de736637b73f047dcb2abaa5854e313cfb99697ae4a163ea4f23\": container with ID starting with bff8a1887121de736637b73f047dcb2abaa5854e313cfb99697ae4a163ea4f23 not found: ID does not exist"
Sep 16 12:22:47.711739 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.711675    2579 scope.go:117] "RemoveContainer" containerID="07290b5772f7eecfe766e21ee9ad8875d87f108e3c87ff96ef26db0c36b6ad30"
Sep 16 12:22:47.712086 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:22:47.712050    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07290b5772f7eecfe766e21ee9ad8875d87f108e3c87ff96ef26db0c36b6ad30\": container with ID starting with 07290b5772f7eecfe766e21ee9ad8875d87f108e3c87ff96ef26db0c36b6ad30 not found: ID does not exist" containerID="07290b5772f7eecfe766e21ee9ad8875d87f108e3c87ff96ef26db0c36b6ad30"
Sep 16 12:22:47.712175 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.712092    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07290b5772f7eecfe766e21ee9ad8875d87f108e3c87ff96ef26db0c36b6ad30"} err="failed to get container status \"07290b5772f7eecfe766e21ee9ad8875d87f108e3c87ff96ef26db0c36b6ad30\": rpc error: code = NotFound desc = could not find container \"07290b5772f7eecfe766e21ee9ad8875d87f108e3c87ff96ef26db0c36b6ad30\": container with ID starting with 07290b5772f7eecfe766e21ee9ad8875d87f108e3c87ff96ef26db0c36b6ad30 not found: ID does not exist"
Sep 16 12:22:47.714693 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.714672    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"]
Sep 16 12:22:47.718742 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:47.718721    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-c2aae110-openshift-default-68489dd8c6-ptxhw"]
Sep 16 12:22:48.781931 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:48.781897    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f17ee56-1a01-4732-8359-4dea9620fc8b" path="/var/lib/kubelet/pods/2f17ee56-1a01-4732-8359-4dea9620fc8b/volumes"
Sep 16 12:22:56.154503 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:22:56.154466    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:23:20.643613 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:20.643573    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"]
Sep 16 12:23:20.644222 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:20.643918    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw" podUID="ae2dd3b3-17b3-4556-9340-d4a69042ab8a" containerName="istio-proxy" containerID="cri-o://6be0d0cc8c814369d8c33ddaab732b4fea81c383f456e5c589617cd90f5414ee" gracePeriod=30
Sep 16 12:23:21.354417 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.354382    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"]
Sep 16 12:23:21.354823 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.354809    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f17ee56-1a01-4732-8359-4dea9620fc8b" containerName="istio-proxy"
Sep 16 12:23:21.354876 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.354824    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f17ee56-1a01-4732-8359-4dea9620fc8b" containerName="istio-proxy"
Sep 16 12:23:21.354876 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.354834    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" containerName="istio-proxy"
Sep 16 12:23:21.354876 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.354840    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" containerName="istio-proxy"
Sep 16 12:23:21.354876 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.354845    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f17ee56-1a01-4732-8359-4dea9620fc8b" containerName="istio-proxy"
Sep 16 12:23:21.354876 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.354851    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f17ee56-1a01-4732-8359-4dea9620fc8b" containerName="istio-proxy"
Sep 16 12:23:21.354876 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.354858    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f0d8036-7963-408b-9782-73ff346922a3" containerName="maas-api"
Sep 16 12:23:21.354876 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.354863    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f0d8036-7963-408b-9782-73ff346922a3" containerName="maas-api"
Sep 16 12:23:21.355087 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.354940    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d8af3a6-2d34-4b3f-ab9f-676c8450ab22" containerName="istio-proxy"
Sep 16 12:23:21.355087 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.354948    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f17ee56-1a01-4732-8359-4dea9620fc8b" containerName="istio-proxy"
Sep 16 12:23:21.355087 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.354958    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f0d8036-7963-408b-9782-73ff346922a3" containerName="maas-api"
Sep 16 12:23:21.357498 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.357474    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.360044 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.360015    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-f1730c-openshift-default-dockercfg-hrnsw\""
Sep 16 12:23:21.370747 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.370719    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"]
Sep 16 12:23:21.422054 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.422016    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-istio-data\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.422260 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.422077    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/756d5144-227c-42ad-948f-72abbf8c8fa0-istio-podinfo\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.422260 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.422106    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rlvd\" (UniqueName: \"kubernetes.io/projected/756d5144-227c-42ad-948f-72abbf8c8fa0-kube-api-access-5rlvd\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.422260 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.422128    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-credential-socket\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.422260 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.422206    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-istio-envoy\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.422462 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.422273    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-workload-socket\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.422462 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.422300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/756d5144-227c-42ad-948f-72abbf8c8fa0-istiod-ca-cert\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.422462 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.422338    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/756d5144-227c-42ad-948f-72abbf8c8fa0-istio-token\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.422462 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.422418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-workload-certs\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.523236 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.523200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-workload-socket\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.523236 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.523237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/756d5144-227c-42ad-948f-72abbf8c8fa0-istiod-ca-cert\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.523482 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.523261    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/756d5144-227c-42ad-948f-72abbf8c8fa0-istio-token\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.523482 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.523292    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-workload-certs\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.523482 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.523410    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-istio-data\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.523703 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.523511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/756d5144-227c-42ad-948f-72abbf8c8fa0-istio-podinfo\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.523703 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.523559    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rlvd\" (UniqueName: \"kubernetes.io/projected/756d5144-227c-42ad-948f-72abbf8c8fa0-kube-api-access-5rlvd\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.523703 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.523588    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-credential-socket\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.523703 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.523651    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-workload-socket\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.523940 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.523708    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-workload-certs\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.523940 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.523713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-istio-envoy\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.524051 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.523958    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-credential-socket\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.524119 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.524062    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-istio-data\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.524255 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.524227    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/756d5144-227c-42ad-948f-72abbf8c8fa0-istiod-ca-cert\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.525880 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.525858    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-istio-envoy\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.526012 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.525996    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/756d5144-227c-42ad-948f-72abbf8c8fa0-istio-podinfo\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.534356 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.534332    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/756d5144-227c-42ad-948f-72abbf8c8fa0-istio-token\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.534991 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.534967    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rlvd\" (UniqueName: \"kubernetes.io/projected/756d5144-227c-42ad-948f-72abbf8c8fa0-kube-api-access-5rlvd\") pod \"e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") " pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.668627 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.668538    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:21.942336 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.942309    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"]
Sep 16 12:23:21.943852 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:23:21.943823    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod756d5144_227c_42ad_948f_72abbf8c8fa0.slice/crio-7d4cd71f4a9bd68bccc0e0a9b134967f2fe2c57837732483fd66070b787cb330 WatchSource:0}: Error finding container 7d4cd71f4a9bd68bccc0e0a9b134967f2fe2c57837732483fd66070b787cb330: Status 404 returned error can't find the container with id 7d4cd71f4a9bd68bccc0e0a9b134967f2fe2c57837732483fd66070b787cb330
Sep 16 12:23:21.945958 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.945926    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:23:21.946042 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.945990    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:23:21.946042 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:21.946019    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:23:22.902022 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:22.901982    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg" event={"ID":"756d5144-227c-42ad-948f-72abbf8c8fa0","Type":"ContainerStarted","Data":"4ea024ea239d155437d088b955e4cb1f5ff36569ce85b4b2891f3eac7964d4d8"}
Sep 16 12:23:22.902474 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:22.902043    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg" event={"ID":"756d5144-227c-42ad-948f-72abbf8c8fa0","Type":"ContainerStarted","Data":"7d4cd71f4a9bd68bccc0e0a9b134967f2fe2c57837732483fd66070b787cb330"}
Sep 16 12:23:22.921434 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:22.921386    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg" podStartSLOduration=1.921371736 podStartE2EDuration="1.921371736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:23:22.91939892 +0000 UTC m=+1196.620470248" watchObservedRunningTime="2026-09-16 12:23:22.921371736 +0000 UTC m=+1196.622443061"
Sep 16 12:23:23.669531 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:23.669491    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:23.674584 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:23.674554    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:23.907969 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:23.907942    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:23.909330 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:23.909310    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:25.261878 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.261839    2579 patch_prober.go:28] interesting pod/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 12:23:25.262260 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.261907    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw" podUID="ae2dd3b3-17b3-4556-9340-d4a69042ab8a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:23:25.891858 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.891827    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:23:25.923079 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.922990    2579 generic.go:356] "Generic (PLEG): container finished" podID="ae2dd3b3-17b3-4556-9340-d4a69042ab8a" containerID="6be0d0cc8c814369d8c33ddaab732b4fea81c383f456e5c589617cd90f5414ee" exitCode=0
Sep 16 12:23:25.923079 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.923036    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"
Sep 16 12:23:25.923262 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.923074    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw" event={"ID":"ae2dd3b3-17b3-4556-9340-d4a69042ab8a","Type":"ContainerDied","Data":"6be0d0cc8c814369d8c33ddaab732b4fea81c383f456e5c589617cd90f5414ee"}
Sep 16 12:23:25.923262 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.923129    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw" event={"ID":"ae2dd3b3-17b3-4556-9340-d4a69042ab8a","Type":"ContainerDied","Data":"8801e8cce47a5ca5808734cb2add05946534661ffe50eae05427c181f78f3bf1"}
Sep 16 12:23:25.923262 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.923160    2579 scope.go:117] "RemoveContainer" containerID="6be0d0cc8c814369d8c33ddaab732b4fea81c383f456e5c589617cd90f5414ee"
Sep 16 12:23:25.934555 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.934533    2579 scope.go:117] "RemoveContainer" containerID="3cd18fed7a719a18f3166d34d1b3c5f019b96441726967f3627146270fb2928c"
Sep 16 12:23:25.947990 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.947932    2579 scope.go:117] "RemoveContainer" containerID="6be0d0cc8c814369d8c33ddaab732b4fea81c383f456e5c589617cd90f5414ee"
Sep 16 12:23:25.948332 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:23:25.948310    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6be0d0cc8c814369d8c33ddaab732b4fea81c383f456e5c589617cd90f5414ee\": container with ID starting with 6be0d0cc8c814369d8c33ddaab732b4fea81c383f456e5c589617cd90f5414ee not found: ID does not exist" containerID="6be0d0cc8c814369d8c33ddaab732b4fea81c383f456e5c589617cd90f5414ee"
Sep 16 12:23:25.948429 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.948340    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6be0d0cc8c814369d8c33ddaab732b4fea81c383f456e5c589617cd90f5414ee"} err="failed to get container status \"6be0d0cc8c814369d8c33ddaab732b4fea81c383f456e5c589617cd90f5414ee\": rpc error: code = NotFound desc = could not find container \"6be0d0cc8c814369d8c33ddaab732b4fea81c383f456e5c589617cd90f5414ee\": container with ID starting with 6be0d0cc8c814369d8c33ddaab732b4fea81c383f456e5c589617cd90f5414ee not found: ID does not exist"
Sep 16 12:23:25.948429 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.948358    2579 scope.go:117] "RemoveContainer" containerID="3cd18fed7a719a18f3166d34d1b3c5f019b96441726967f3627146270fb2928c"
Sep 16 12:23:25.948746 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:23:25.948642    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cd18fed7a719a18f3166d34d1b3c5f019b96441726967f3627146270fb2928c\": container with ID starting with 3cd18fed7a719a18f3166d34d1b3c5f019b96441726967f3627146270fb2928c not found: ID does not exist" containerID="3cd18fed7a719a18f3166d34d1b3c5f019b96441726967f3627146270fb2928c"
Sep 16 12:23:25.948746 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.948693    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cd18fed7a719a18f3166d34d1b3c5f019b96441726967f3627146270fb2928c"} err="failed to get container status \"3cd18fed7a719a18f3166d34d1b3c5f019b96441726967f3627146270fb2928c\": rpc error: code = NotFound desc = could not find container \"3cd18fed7a719a18f3166d34d1b3c5f019b96441726967f3627146270fb2928c\": container with ID starting with 3cd18fed7a719a18f3166d34d1b3c5f019b96441726967f3627146270fb2928c not found: ID does not exist"
Sep 16 12:23:25.967996 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.967974    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istio-token\") pod \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") "
Sep 16 12:23:25.968100 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.968030    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istio-podinfo\") pod \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") "
Sep 16 12:23:25.968100 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.968074    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-credential-socket\") pod \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") "
Sep 16 12:23:25.968275 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.968253    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5g4kj\" (UniqueName: \"kubernetes.io/projected/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-kube-api-access-5g4kj\") pod \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") "
Sep 16 12:23:25.968328 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.968304    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-workload-certs\") pod \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") "
Sep 16 12:23:25.968328 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.968315    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ae2dd3b3-17b3-4556-9340-d4a69042ab8a" (UID: "ae2dd3b3-17b3-4556-9340-d4a69042ab8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:23:25.968438 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.968338    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istio-envoy\") pod \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") "
Sep 16 12:23:25.968438 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.968378    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-workload-socket\") pod \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") "
Sep 16 12:23:25.968438 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.968406    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istiod-ca-cert\") pod \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") "
Sep 16 12:23:25.968438 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.968433    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istio-data\") pod \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\" (UID: \"ae2dd3b3-17b3-4556-9340-d4a69042ab8a\") "
Sep 16 12:23:25.968687 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.968501    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ae2dd3b3-17b3-4556-9340-d4a69042ab8a" (UID: "ae2dd3b3-17b3-4556-9340-d4a69042ab8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:23:25.968687 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.968608    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ae2dd3b3-17b3-4556-9340-d4a69042ab8a" (UID: "ae2dd3b3-17b3-4556-9340-d4a69042ab8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:23:25.969254 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.968940    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-credential-socket\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:23:25.969254 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.968964    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-workload-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:23:25.969254 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.968979    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-workload-socket\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:23:25.969254 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.969111    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istio-data" (OuterVolumeSpecName: "istio-data") pod "ae2dd3b3-17b3-4556-9340-d4a69042ab8a" (UID: "ae2dd3b3-17b3-4556-9340-d4a69042ab8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:23:25.969254 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.969156    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ae2dd3b3-17b3-4556-9340-d4a69042ab8a" (UID: "ae2dd3b3-17b3-4556-9340-d4a69042ab8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:23:25.970299 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.970276    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istio-token" (OuterVolumeSpecName: "istio-token") pod "ae2dd3b3-17b3-4556-9340-d4a69042ab8a" (UID: "ae2dd3b3-17b3-4556-9340-d4a69042ab8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:23:25.970465 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.970450    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-kube-api-access-5g4kj" (OuterVolumeSpecName: "kube-api-access-5g4kj") pod "ae2dd3b3-17b3-4556-9340-d4a69042ab8a" (UID: "ae2dd3b3-17b3-4556-9340-d4a69042ab8a"). InnerVolumeSpecName "kube-api-access-5g4kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:23:25.970707 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.970685    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ae2dd3b3-17b3-4556-9340-d4a69042ab8a" (UID: "ae2dd3b3-17b3-4556-9340-d4a69042ab8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:23:25.970803 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:25.970684    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ae2dd3b3-17b3-4556-9340-d4a69042ab8a" (UID: "ae2dd3b3-17b3-4556-9340-d4a69042ab8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:23:26.070350 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:26.070317    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istio-token\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:23:26.070350 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:26.070345    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istio-podinfo\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:23:26.070350 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:26.070356    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g4kj\" (UniqueName: \"kubernetes.io/projected/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-kube-api-access-5g4kj\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:23:26.070579 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:26.070366    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istio-envoy\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:23:26.070579 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:26.070374    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istiod-ca-cert\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:23:26.070579 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:26.070381    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae2dd3b3-17b3-4556-9340-d4a69042ab8a-istio-data\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:23:26.245849 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:26.245816    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"]
Sep 16 12:23:26.253311 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:26.253285    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-00a141-openshift-default-5fbf7f84cb-cqvsw"]
Sep 16 12:23:26.782496 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:26.782464    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae2dd3b3-17b3-4556-9340-d4a69042ab8a" path="/var/lib/kubelet/pods/ae2dd3b3-17b3-4556-9340-d4a69042ab8a/volumes"
Sep 16 12:23:27.817232 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:27.817204    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb"]
Sep 16 12:23:27.817610 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:27.817597    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae2dd3b3-17b3-4556-9340-d4a69042ab8a" containerName="istio-proxy"
Sep 16 12:23:27.817675 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:27.817612    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae2dd3b3-17b3-4556-9340-d4a69042ab8a" containerName="istio-proxy"
Sep 16 12:23:27.817740 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:27.817704    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae2dd3b3-17b3-4556-9340-d4a69042ab8a" containerName="istio-proxy"
Sep 16 12:23:27.817740 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:27.817714    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae2dd3b3-17b3-4556-9340-d4a69042ab8a" containerName="istio-proxy"
Sep 16 12:23:27.817740 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:27.817722    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f17ee56-1a01-4732-8359-4dea9620fc8b" containerName="istio-proxy"
Sep 16 12:23:27.829178 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:27.829137    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb"
Sep 16 12:23:27.832176 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:27.832144    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-f1730c-serving-cert\""
Sep 16 12:23:27.832429 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:27.832163    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 12:23:27.832545 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:27.832163    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:23:27.832545 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:27.832167    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4gkmq\""
Sep 16 12:23:27.832691 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:27.832230    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 12:23:27.832859 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:27.832838    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb"]
Sep 16 12:23:27.888734 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:27.888708    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pwdq\" (UniqueName: \"kubernetes.io/projected/aa760fc8-008d-40f6-b924-021e43a69199-kube-api-access-7pwdq\") pod \"maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb\" (UID: \"aa760fc8-008d-40f6-b924-021e43a69199\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb"
Sep 16 12:23:27.888896 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:27.888759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa760fc8-008d-40f6-b924-021e43a69199-openshift-service-ca\") pod \"maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb\" (UID: \"aa760fc8-008d-40f6-b924-021e43a69199\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb"
Sep 16 12:23:27.888896 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:27.888787    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa760fc8-008d-40f6-b924-021e43a69199-metrics-certs\") pod \"maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb\" (UID: \"aa760fc8-008d-40f6-b924-021e43a69199\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb"
Sep 16 12:23:27.888896 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:27.888870    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa760fc8-008d-40f6-b924-021e43a69199-maas-api-tls\") pod \"maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb\" (UID: \"aa760fc8-008d-40f6-b924-021e43a69199\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb"
Sep 16 12:23:27.990323 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:27.990290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pwdq\" (UniqueName: \"kubernetes.io/projected/aa760fc8-008d-40f6-b924-021e43a69199-kube-api-access-7pwdq\") pod \"maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb\" (UID: \"aa760fc8-008d-40f6-b924-021e43a69199\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb"
Sep 16 12:23:27.990502 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:27.990336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa760fc8-008d-40f6-b924-021e43a69199-openshift-service-ca\") pod \"maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb\" (UID: \"aa760fc8-008d-40f6-b924-021e43a69199\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb"
Sep 16 12:23:27.990502 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:27.990368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa760fc8-008d-40f6-b924-021e43a69199-metrics-certs\") pod \"maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb\" (UID: \"aa760fc8-008d-40f6-b924-021e43a69199\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb"
Sep 16 12:23:27.990502 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:27.990430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa760fc8-008d-40f6-b924-021e43a69199-maas-api-tls\") pod \"maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb\" (UID: \"aa760fc8-008d-40f6-b924-021e43a69199\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb"
Sep 16 12:23:27.990671 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:23:27.990530    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-serving-cert: secret "maas-api-e2e-explicit-f1730c-serving-cert" not found
Sep 16 12:23:27.990671 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:23:27.990595    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa760fc8-008d-40f6-b924-021e43a69199-maas-api-tls podName:aa760fc8-008d-40f6-b924-021e43a69199 nodeName:}" failed. No retries permitted until 2026-09-16 12:23:28.490575814 +0000 UTC m=+1202.191647119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aa760fc8-008d-40f6-b924-021e43a69199-maas-api-tls") pod "maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb" (UID: "aa760fc8-008d-40f6-b924-021e43a69199") : secret "maas-api-e2e-explicit-f1730c-serving-cert" not found
Sep 16 12:23:27.991060 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:27.991040    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa760fc8-008d-40f6-b924-021e43a69199-openshift-service-ca\") pod \"maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb\" (UID: \"aa760fc8-008d-40f6-b924-021e43a69199\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb"
Sep 16 12:23:27.993050 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:27.993026    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa760fc8-008d-40f6-b924-021e43a69199-metrics-certs\") pod \"maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb\" (UID: \"aa760fc8-008d-40f6-b924-021e43a69199\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb"
Sep 16 12:23:27.999416 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:27.999397    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pwdq\" (UniqueName: \"kubernetes.io/projected/aa760fc8-008d-40f6-b924-021e43a69199-kube-api-access-7pwdq\") pod \"maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb\" (UID: \"aa760fc8-008d-40f6-b924-021e43a69199\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb"
Sep 16 12:23:28.496012 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:28.495972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa760fc8-008d-40f6-b924-021e43a69199-maas-api-tls\") pod \"maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb\" (UID: \"aa760fc8-008d-40f6-b924-021e43a69199\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb"
Sep 16 12:23:28.498310 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:28.498279    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa760fc8-008d-40f6-b924-021e43a69199-maas-api-tls\") pod \"maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb\" (UID: \"aa760fc8-008d-40f6-b924-021e43a69199\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb"
Sep 16 12:23:28.742095 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:28.742064    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb"
Sep 16 12:23:29.013709 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:29.013680    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb"]
Sep 16 12:23:29.015040 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:23:29.015010    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa760fc8_008d_40f6_b924_021e43a69199.slice/crio-0bb1af5aac91363df7a4b20c48648a8fd7f54101e6d9f6f308236e7714225dee WatchSource:0}: Error finding container 0bb1af5aac91363df7a4b20c48648a8fd7f54101e6d9f6f308236e7714225dee: Status 404 returned error can't find the container with id 0bb1af5aac91363df7a4b20c48648a8fd7f54101e6d9f6f308236e7714225dee
Sep 16 12:23:29.952382 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:29.952338    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb" event={"ID":"aa760fc8-008d-40f6-b924-021e43a69199","Type":"ContainerStarted","Data":"66f28b555b55bf8f81824b7285d30da5672dffd46e84a838528b830ce9feb957"}
Sep 16 12:23:29.952554 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:29.952406    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb" event={"ID":"aa760fc8-008d-40f6-b924-021e43a69199","Type":"ContainerStarted","Data":"0bb1af5aac91363df7a4b20c48648a8fd7f54101e6d9f6f308236e7714225dee"}
Sep 16 12:23:29.952554 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:29.952434    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb"
Sep 16 12:23:29.971178 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:29.971131    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb" podStartSLOduration=2.5330937479999998 podStartE2EDuration="2.971115694s" totalImagesPullingTime="438.021946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:23:29.969809901 +0000 UTC m=+1203.670881278" watchObservedRunningTime="2026-09-16 12:23:29.971115694 +0000 UTC m=+1203.672187019"
Sep 16 12:23:35.964182 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:35.964151    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb"
Sep 16 12:23:36.768818 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:36.768780    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:23:44.824109 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:44.824073    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"]
Sep 16 12:23:44.824567 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:44.824478    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae2dd3b3-17b3-4556-9340-d4a69042ab8a" containerName="istio-proxy"
Sep 16 12:23:44.824567 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:44.824489    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae2dd3b3-17b3-4556-9340-d4a69042ab8a" containerName="istio-proxy"
Sep 16 12:23:44.827192 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:44.827172    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:23:44.829951 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:44.829904    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-739ee7\"/\"openshift-service-ca.crt\""
Sep 16 12:23:44.829951 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:44.829910    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-739ee7\"/\"kube-root-ca.crt\""
Sep 16 12:23:44.832294 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:44.831913    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-739ee7\"/\"default-dockercfg-cwc4z\""
Sep 16 12:23:44.832294 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:44.832224    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-739ee7\"/\"premium-w0-739ee7-kserve-self-signed-certs\""
Sep 16 12:23:44.837444 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:44.837424    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"]
Sep 16 12:23:44.967280 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:44.967247    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d2173d2-5881-4e76-8de6-548d486111db-home\") pod \"premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh\" (UID: \"9d2173d2-5881-4e76-8de6-548d486111db\") " pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:23:44.967469 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:44.967294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d2173d2-5881-4e76-8de6-548d486111db-dshm\") pod \"premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh\" (UID: \"9d2173d2-5881-4e76-8de6-548d486111db\") " pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:23:44.967469 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:44.967333    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9prbd\" (UniqueName: \"kubernetes.io/projected/9d2173d2-5881-4e76-8de6-548d486111db-kube-api-access-9prbd\") pod \"premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh\" (UID: \"9d2173d2-5881-4e76-8de6-548d486111db\") " pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:23:44.967591 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:44.967530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2173d2-5881-4e76-8de6-548d486111db-tmp-dir\") pod \"premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh\" (UID: \"9d2173d2-5881-4e76-8de6-548d486111db\") " pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:23:44.967689 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:44.967640    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d2173d2-5881-4e76-8de6-548d486111db-tls-certs\") pod \"premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh\" (UID: \"9d2173d2-5881-4e76-8de6-548d486111db\") " pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:23:44.967762 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:44.967705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d2173d2-5881-4e76-8de6-548d486111db-model-cache\") pod \"premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh\" (UID: \"9d2173d2-5881-4e76-8de6-548d486111db\") " pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:23:45.069164 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:45.069123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d2173d2-5881-4e76-8de6-548d486111db-home\") pod \"premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh\" (UID: \"9d2173d2-5881-4e76-8de6-548d486111db\") " pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:23:45.069352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:45.069172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d2173d2-5881-4e76-8de6-548d486111db-dshm\") pod \"premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh\" (UID: \"9d2173d2-5881-4e76-8de6-548d486111db\") " pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:23:45.069352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:45.069202    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9prbd\" (UniqueName: \"kubernetes.io/projected/9d2173d2-5881-4e76-8de6-548d486111db-kube-api-access-9prbd\") pod \"premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh\" (UID: \"9d2173d2-5881-4e76-8de6-548d486111db\") " pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:23:45.069352 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:45.069284    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2173d2-5881-4e76-8de6-548d486111db-tmp-dir\") pod \"premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh\" (UID: \"9d2173d2-5881-4e76-8de6-548d486111db\") " pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:23:45.069520 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:45.069356    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d2173d2-5881-4e76-8de6-548d486111db-tls-certs\") pod \"premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh\" (UID: \"9d2173d2-5881-4e76-8de6-548d486111db\") " pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:23:45.069520 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:45.069380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d2173d2-5881-4e76-8de6-548d486111db-model-cache\") pod \"premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh\" (UID: \"9d2173d2-5881-4e76-8de6-548d486111db\") " pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:23:45.069520 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:45.069479    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d2173d2-5881-4e76-8de6-548d486111db-home\") pod \"premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh\" (UID: \"9d2173d2-5881-4e76-8de6-548d486111db\") " pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:23:45.069733 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:45.069714    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2173d2-5881-4e76-8de6-548d486111db-tmp-dir\") pod \"premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh\" (UID: \"9d2173d2-5881-4e76-8de6-548d486111db\") " pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:23:45.069786 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:45.069715    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d2173d2-5881-4e76-8de6-548d486111db-model-cache\") pod \"premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh\" (UID: \"9d2173d2-5881-4e76-8de6-548d486111db\") " pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:23:45.071354 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:45.071327    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d2173d2-5881-4e76-8de6-548d486111db-dshm\") pod \"premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh\" (UID: \"9d2173d2-5881-4e76-8de6-548d486111db\") " pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:23:45.071817 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:45.071799    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d2173d2-5881-4e76-8de6-548d486111db-tls-certs\") pod \"premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh\" (UID: \"9d2173d2-5881-4e76-8de6-548d486111db\") " pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:23:45.081360 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:45.081335    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9prbd\" (UniqueName: \"kubernetes.io/projected/9d2173d2-5881-4e76-8de6-548d486111db-kube-api-access-9prbd\") pod \"premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh\" (UID: \"9d2173d2-5881-4e76-8de6-548d486111db\") " pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:23:45.141104 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:45.141065    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:23:45.417194 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:45.417164    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"]
Sep 16 12:23:45.418286 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:23:45.418261    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d2173d2_5881_4e76_8de6_548d486111db.slice/crio-addbf7a63fe2a07ba31a12ff473754a53f7772bf09fefb98e6a00938b454f1b7 WatchSource:0}: Error finding container addbf7a63fe2a07ba31a12ff473754a53f7772bf09fefb98e6a00938b454f1b7: Status 404 returned error can't find the container with id addbf7a63fe2a07ba31a12ff473754a53f7772bf09fefb98e6a00938b454f1b7
Sep 16 12:23:46.052373 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:46.052335    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh" event={"ID":"9d2173d2-5881-4e76-8de6-548d486111db","Type":"ContainerStarted","Data":"7d2dc0c293c23a239d30f2e70c075ed465330ebf01fce1872edc7d1570ccfd55"}
Sep 16 12:23:46.052373 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:46.052380    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh" event={"ID":"9d2173d2-5881-4e76-8de6-548d486111db","Type":"ContainerStarted","Data":"addbf7a63fe2a07ba31a12ff473754a53f7772bf09fefb98e6a00938b454f1b7"}
Sep 16 12:23:46.072265 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:46.072201    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh" podStartSLOduration=2.072180405 podStartE2EDuration="2.072180405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:23:46.069622392 +0000 UTC m=+1219.770693729" watchObservedRunningTime="2026-09-16 12:23:46.072180405 +0000 UTC m=+1219.773251733"
Sep 16 12:23:47.059801 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:47.059763    2579 generic.go:356] "Generic (PLEG): container finished" podID="756d5144-227c-42ad-948f-72abbf8c8fa0" containerID="4ea024ea239d155437d088b955e4cb1f5ff36569ce85b4b2891f3eac7964d4d8" exitCode=0
Sep 16 12:23:47.060290 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:47.059828    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg" event={"ID":"756d5144-227c-42ad-948f-72abbf8c8fa0","Type":"ContainerDied","Data":"4ea024ea239d155437d088b955e4cb1f5ff36569ce85b4b2891f3eac7964d4d8"}
Sep 16 12:23:47.060410 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:47.060385    2579 scope.go:117] "RemoveContainer" containerID="4ea024ea239d155437d088b955e4cb1f5ff36569ce85b4b2891f3eac7964d4d8"
Sep 16 12:23:47.061454 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:47.061426    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:23:47.061535 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:47.061512    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:23:47.061589 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:47.061559    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:23:48.067744 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:48.067709    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg" event={"ID":"756d5144-227c-42ad-948f-72abbf8c8fa0","Type":"ContainerStarted","Data":"cc14db5de54d7058e83957b32b1ae744fc122fb2d222f9158e6f06e767ee9131"}
Sep 16 12:23:48.669183 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:48.669133    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:48.669367 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:48.669343    2579 patch_prober.go:28] interesting pod/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 16 12:23:48.669418 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:48.669397    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg" podUID="756d5144-227c-42ad-948f-72abbf8c8fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 16 12:23:49.669447 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:49.669411    2579 patch_prober.go:28] interesting pod/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 16 12:23:49.669872 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:49.669470    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg" podUID="756d5144-227c-42ad-948f-72abbf8c8fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 16 12:23:50.669701 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:50.669642    2579 patch_prober.go:28] interesting pod/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 16 12:23:50.670096 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:50.669752    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg" podUID="756d5144-227c-42ad-948f-72abbf8c8fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 16 12:23:51.373147 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:51.373107    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:23:51.669735 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:51.669648    2579 patch_prober.go:28] interesting pod/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 16 12:23:51.669735 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:51.669717    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg" podUID="756d5144-227c-42ad-948f-72abbf8c8fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 16 12:23:52.669726 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:52.669683    2579 patch_prober.go:28] interesting pod/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 16 12:23:52.670253 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:52.669770    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg" podUID="756d5144-227c-42ad-948f-72abbf8c8fa0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 16 12:23:53.673526 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:53.673493    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:53.673931 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:53.673843    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:53.674672 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:53.674639    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:23:55.141525 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:55.141490    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:23:55.142042 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:55.141539    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:23:55.154166 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:55.154143    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:23:56.136243 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:23:56.136210    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:24:12.859517 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:12.859477    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:24:26.126020 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.125985    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"]
Sep 16 12:24:26.128943 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.128921    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:24:26.132197 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.132179    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ef3317\"/\"kube-root-ca.crt\""
Sep 16 12:24:26.132322 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.132304    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ef3317\"/\"openshift-service-ca.crt\""
Sep 16 12:24:26.133435 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.133415    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ef3317\"/\"test-model-ef3317-kserve-self-signed-certs\""
Sep 16 12:24:26.133512 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.133472    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ef3317\"/\"default-dockercfg-xt6z2\""
Sep 16 12:24:26.140775 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.140750    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"]
Sep 16 12:24:26.255368 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.255324    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a40af25e-97c6-4425-8256-e07f985836b5-tls-certs\") pod \"test-model-ef3317-kserve-7866bc7f79-bbtnp\" (UID: \"a40af25e-97c6-4425-8256-e07f985836b5\") " pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:24:26.255561 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.255429    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmmvt\" (UniqueName: \"kubernetes.io/projected/a40af25e-97c6-4425-8256-e07f985836b5-kube-api-access-cmmvt\") pod \"test-model-ef3317-kserve-7866bc7f79-bbtnp\" (UID: \"a40af25e-97c6-4425-8256-e07f985836b5\") " pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:24:26.255561 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.255456    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a40af25e-97c6-4425-8256-e07f985836b5-home\") pod \"test-model-ef3317-kserve-7866bc7f79-bbtnp\" (UID: \"a40af25e-97c6-4425-8256-e07f985836b5\") " pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:24:26.255561 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.255497    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a40af25e-97c6-4425-8256-e07f985836b5-model-cache\") pod \"test-model-ef3317-kserve-7866bc7f79-bbtnp\" (UID: \"a40af25e-97c6-4425-8256-e07f985836b5\") " pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:24:26.255561 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.255530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a40af25e-97c6-4425-8256-e07f985836b5-dshm\") pod \"test-model-ef3317-kserve-7866bc7f79-bbtnp\" (UID: \"a40af25e-97c6-4425-8256-e07f985836b5\") " pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:24:26.255561 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.255560    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a40af25e-97c6-4425-8256-e07f985836b5-tmp-dir\") pod \"test-model-ef3317-kserve-7866bc7f79-bbtnp\" (UID: \"a40af25e-97c6-4425-8256-e07f985836b5\") " pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:24:26.356460 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.356426    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a40af25e-97c6-4425-8256-e07f985836b5-tls-certs\") pod \"test-model-ef3317-kserve-7866bc7f79-bbtnp\" (UID: \"a40af25e-97c6-4425-8256-e07f985836b5\") " pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:24:26.356628 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.356496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmmvt\" (UniqueName: \"kubernetes.io/projected/a40af25e-97c6-4425-8256-e07f985836b5-kube-api-access-cmmvt\") pod \"test-model-ef3317-kserve-7866bc7f79-bbtnp\" (UID: \"a40af25e-97c6-4425-8256-e07f985836b5\") " pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:24:26.356628 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.356517    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a40af25e-97c6-4425-8256-e07f985836b5-home\") pod \"test-model-ef3317-kserve-7866bc7f79-bbtnp\" (UID: \"a40af25e-97c6-4425-8256-e07f985836b5\") " pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:24:26.356628 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.356543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a40af25e-97c6-4425-8256-e07f985836b5-model-cache\") pod \"test-model-ef3317-kserve-7866bc7f79-bbtnp\" (UID: \"a40af25e-97c6-4425-8256-e07f985836b5\") " pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:24:26.356628 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.356566    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a40af25e-97c6-4425-8256-e07f985836b5-dshm\") pod \"test-model-ef3317-kserve-7866bc7f79-bbtnp\" (UID: \"a40af25e-97c6-4425-8256-e07f985836b5\") " pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:24:26.356628 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.356588    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a40af25e-97c6-4425-8256-e07f985836b5-tmp-dir\") pod \"test-model-ef3317-kserve-7866bc7f79-bbtnp\" (UID: \"a40af25e-97c6-4425-8256-e07f985836b5\") " pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:24:26.357007 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.356977    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a40af25e-97c6-4425-8256-e07f985836b5-home\") pod \"test-model-ef3317-kserve-7866bc7f79-bbtnp\" (UID: \"a40af25e-97c6-4425-8256-e07f985836b5\") " pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:24:26.357007 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.357000    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a40af25e-97c6-4425-8256-e07f985836b5-tmp-dir\") pod \"test-model-ef3317-kserve-7866bc7f79-bbtnp\" (UID: \"a40af25e-97c6-4425-8256-e07f985836b5\") " pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:24:26.357170 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.357034    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a40af25e-97c6-4425-8256-e07f985836b5-model-cache\") pod \"test-model-ef3317-kserve-7866bc7f79-bbtnp\" (UID: \"a40af25e-97c6-4425-8256-e07f985836b5\") " pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:24:26.358845 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.358827    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a40af25e-97c6-4425-8256-e07f985836b5-dshm\") pod \"test-model-ef3317-kserve-7866bc7f79-bbtnp\" (UID: \"a40af25e-97c6-4425-8256-e07f985836b5\") " pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:24:26.359131 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.359111    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a40af25e-97c6-4425-8256-e07f985836b5-tls-certs\") pod \"test-model-ef3317-kserve-7866bc7f79-bbtnp\" (UID: \"a40af25e-97c6-4425-8256-e07f985836b5\") " pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:24:26.365137 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.365110    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmmvt\" (UniqueName: \"kubernetes.io/projected/a40af25e-97c6-4425-8256-e07f985836b5-kube-api-access-cmmvt\") pod \"test-model-ef3317-kserve-7866bc7f79-bbtnp\" (UID: \"a40af25e-97c6-4425-8256-e07f985836b5\") " pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:24:26.439798 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.439707    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:24:26.716559 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:26.716534    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"]
Sep 16 12:24:26.717852 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:24:26.717821    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda40af25e_97c6_4425_8256_e07f985836b5.slice/crio-178026a625e71fc89d6860b04c2303cde6041196f92c40e331a95b7f071d6ce9 WatchSource:0}: Error finding container 178026a625e71fc89d6860b04c2303cde6041196f92c40e331a95b7f071d6ce9: Status 404 returned error can't find the container with id 178026a625e71fc89d6860b04c2303cde6041196f92c40e331a95b7f071d6ce9
Sep 16 12:24:27.314497 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:27.314462    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp" event={"ID":"a40af25e-97c6-4425-8256-e07f985836b5","Type":"ContainerStarted","Data":"66069bedc803016db7cfcecceeea25a6626526dd113c8743949e4370f4aabb51"}
Sep 16 12:24:27.314497 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:27.314504    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp" event={"ID":"a40af25e-97c6-4425-8256-e07f985836b5","Type":"ContainerStarted","Data":"178026a625e71fc89d6860b04c2303cde6041196f92c40e331a95b7f071d6ce9"}
Sep 16 12:24:27.331914 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:27.331860    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp" podStartSLOduration=1.331842526 podStartE2EDuration="1.331842526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:24:27.330118482 +0000 UTC m=+1261.031189833" watchObservedRunningTime="2026-09-16 12:24:27.331842526 +0000 UTC m=+1261.032913852"
Sep 16 12:24:29.336947 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:29.336911    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628"]
Sep 16 12:24:29.339801 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:29.339775    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628"
Sep 16 12:24:29.342725 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:29.342703    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-cadab1d8-serving-cert\""
Sep 16 12:24:29.351214 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:29.351188    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628"]
Sep 16 12:24:29.488417 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:29.488378    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qtfc\" (UniqueName: \"kubernetes.io/projected/a51e67b6-cbf2-42c5-8055-117368c75e1c-kube-api-access-4qtfc\") pod \"maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628\" (UID: \"a51e67b6-cbf2-42c5-8055-117368c75e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628"
Sep 16 12:24:29.488590 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:29.488466    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a51e67b6-cbf2-42c5-8055-117368c75e1c-metrics-certs\") pod \"maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628\" (UID: \"a51e67b6-cbf2-42c5-8055-117368c75e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628"
Sep 16 12:24:29.488590 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:29.488510    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a51e67b6-cbf2-42c5-8055-117368c75e1c-openshift-service-ca\") pod \"maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628\" (UID: \"a51e67b6-cbf2-42c5-8055-117368c75e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628"
Sep 16 12:24:29.488590 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:29.488577    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a51e67b6-cbf2-42c5-8055-117368c75e1c-maas-api-tls\") pod \"maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628\" (UID: \"a51e67b6-cbf2-42c5-8055-117368c75e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628"
Sep 16 12:24:29.589415 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:29.589322    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a51e67b6-cbf2-42c5-8055-117368c75e1c-metrics-certs\") pod \"maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628\" (UID: \"a51e67b6-cbf2-42c5-8055-117368c75e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628"
Sep 16 12:24:29.589415 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:29.589395    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a51e67b6-cbf2-42c5-8055-117368c75e1c-openshift-service-ca\") pod \"maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628\" (UID: \"a51e67b6-cbf2-42c5-8055-117368c75e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628"
Sep 16 12:24:29.589647 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:29.589476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a51e67b6-cbf2-42c5-8055-117368c75e1c-maas-api-tls\") pod \"maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628\" (UID: \"a51e67b6-cbf2-42c5-8055-117368c75e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628"
Sep 16 12:24:29.589647 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:29.589522    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qtfc\" (UniqueName: \"kubernetes.io/projected/a51e67b6-cbf2-42c5-8055-117368c75e1c-kube-api-access-4qtfc\") pod \"maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628\" (UID: \"a51e67b6-cbf2-42c5-8055-117368c75e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628"
Sep 16 12:24:29.590084 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:29.590061    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a51e67b6-cbf2-42c5-8055-117368c75e1c-openshift-service-ca\") pod \"maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628\" (UID: \"a51e67b6-cbf2-42c5-8055-117368c75e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628"
Sep 16 12:24:29.591773 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:29.591750    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a51e67b6-cbf2-42c5-8055-117368c75e1c-metrics-certs\") pod \"maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628\" (UID: \"a51e67b6-cbf2-42c5-8055-117368c75e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628"
Sep 16 12:24:29.591873 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:29.591813    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a51e67b6-cbf2-42c5-8055-117368c75e1c-maas-api-tls\") pod \"maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628\" (UID: \"a51e67b6-cbf2-42c5-8055-117368c75e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628"
Sep 16 12:24:29.598221 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:29.598198    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qtfc\" (UniqueName: \"kubernetes.io/projected/a51e67b6-cbf2-42c5-8055-117368c75e1c-kube-api-access-4qtfc\") pod \"maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628\" (UID: \"a51e67b6-cbf2-42c5-8055-117368c75e1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628"
Sep 16 12:24:29.654412 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:29.654381    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628"
Sep 16 12:24:29.938567 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:29.938538    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628"]
Sep 16 12:24:29.939731 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:24:29.939701    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda51e67b6_cbf2_42c5_8055_117368c75e1c.slice/crio-efe3ee44831644dd74d19872caf60f9c9bb8a05f7272cf4dab15e7861077bc18 WatchSource:0}: Error finding container efe3ee44831644dd74d19872caf60f9c9bb8a05f7272cf4dab15e7861077bc18: Status 404 returned error can't find the container with id efe3ee44831644dd74d19872caf60f9c9bb8a05f7272cf4dab15e7861077bc18
Sep 16 12:24:30.333497 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:30.333461    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628" event={"ID":"a51e67b6-cbf2-42c5-8055-117368c75e1c","Type":"ContainerStarted","Data":"efe3ee44831644dd74d19872caf60f9c9bb8a05f7272cf4dab15e7861077bc18"}
Sep 16 12:24:31.341207 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:31.341171    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628" event={"ID":"a51e67b6-cbf2-42c5-8055-117368c75e1c","Type":"ContainerStarted","Data":"9bbaaa3abcaea59f77baaebb50acd3a74644f571226166e45a62db3ee4cf9819"}
Sep 16 12:24:31.341683 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:31.341236    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628"
Sep 16 12:24:31.358944 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:31.358896    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628" podStartSLOduration=1.912148132 podStartE2EDuration="2.358881116s" totalImagesPullingTime="446.732984ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:24:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:24:31.357274202 +0000 UTC m=+1265.058345528" watchObservedRunningTime="2026-09-16 12:24:31.358881116 +0000 UTC m=+1265.059952482"
Sep 16 12:24:36.440332 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:36.440290    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:24:36.440769 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:36.440354    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:24:36.452692 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:36.452651    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:24:37.352642 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:37.352613    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628"
Sep 16 12:24:37.395458 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:37.395425    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:24:40.175713 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:24:40.175672    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:25:06.365563 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:06.365526    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:25:13.403014 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:13.402975    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628"]
Sep 16 12:25:13.403528 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:13.403251    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628" podUID="a51e67b6-cbf2-42c5-8055-117368c75e1c" containerName="maas-api" containerID="cri-o://9bbaaa3abcaea59f77baaebb50acd3a74644f571226166e45a62db3ee4cf9819" gracePeriod=30
Sep 16 12:25:13.603580 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:13.603547    2579 generic.go:356] "Generic (PLEG): container finished" podID="a51e67b6-cbf2-42c5-8055-117368c75e1c" containerID="9bbaaa3abcaea59f77baaebb50acd3a74644f571226166e45a62db3ee4cf9819" exitCode=0
Sep 16 12:25:13.603773 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:13.603599    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628" event={"ID":"a51e67b6-cbf2-42c5-8055-117368c75e1c","Type":"ContainerDied","Data":"9bbaaa3abcaea59f77baaebb50acd3a74644f571226166e45a62db3ee4cf9819"}
Sep 16 12:25:13.652967 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:13.652939    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628"
Sep 16 12:25:13.731973 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:13.731868    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a51e67b6-cbf2-42c5-8055-117368c75e1c-maas-api-tls\") pod \"a51e67b6-cbf2-42c5-8055-117368c75e1c\" (UID: \"a51e67b6-cbf2-42c5-8055-117368c75e1c\") "
Sep 16 12:25:13.731973 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:13.731972    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qtfc\" (UniqueName: \"kubernetes.io/projected/a51e67b6-cbf2-42c5-8055-117368c75e1c-kube-api-access-4qtfc\") pod \"a51e67b6-cbf2-42c5-8055-117368c75e1c\" (UID: \"a51e67b6-cbf2-42c5-8055-117368c75e1c\") "
Sep 16 12:25:13.732212 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:13.732000    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a51e67b6-cbf2-42c5-8055-117368c75e1c-metrics-certs\") pod \"a51e67b6-cbf2-42c5-8055-117368c75e1c\" (UID: \"a51e67b6-cbf2-42c5-8055-117368c75e1c\") "
Sep 16 12:25:13.732212 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:13.732058    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a51e67b6-cbf2-42c5-8055-117368c75e1c-openshift-service-ca\") pod \"a51e67b6-cbf2-42c5-8055-117368c75e1c\" (UID: \"a51e67b6-cbf2-42c5-8055-117368c75e1c\") "
Sep 16 12:25:13.732543 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:13.732496    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a51e67b6-cbf2-42c5-8055-117368c75e1c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a51e67b6-cbf2-42c5-8055-117368c75e1c" (UID: "a51e67b6-cbf2-42c5-8055-117368c75e1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:25:13.734512 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:13.734468    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a51e67b6-cbf2-42c5-8055-117368c75e1c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a51e67b6-cbf2-42c5-8055-117368c75e1c" (UID: "a51e67b6-cbf2-42c5-8055-117368c75e1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:25:13.734680 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:13.734632    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a51e67b6-cbf2-42c5-8055-117368c75e1c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a51e67b6-cbf2-42c5-8055-117368c75e1c" (UID: "a51e67b6-cbf2-42c5-8055-117368c75e1c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:25:13.734770 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:13.734702    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a51e67b6-cbf2-42c5-8055-117368c75e1c-kube-api-access-4qtfc" (OuterVolumeSpecName: "kube-api-access-4qtfc") pod "a51e67b6-cbf2-42c5-8055-117368c75e1c" (UID: "a51e67b6-cbf2-42c5-8055-117368c75e1c"). InnerVolumeSpecName "kube-api-access-4qtfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:25:13.832932 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:13.832897    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qtfc\" (UniqueName: \"kubernetes.io/projected/a51e67b6-cbf2-42c5-8055-117368c75e1c-kube-api-access-4qtfc\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:25:13.832932 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:13.832926    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a51e67b6-cbf2-42c5-8055-117368c75e1c-metrics-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:25:13.832932 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:13.832936    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a51e67b6-cbf2-42c5-8055-117368c75e1c-openshift-service-ca\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:25:13.832932 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:13.832945    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a51e67b6-cbf2-42c5-8055-117368c75e1c-maas-api-tls\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:25:14.611754 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:14.611714    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628" event={"ID":"a51e67b6-cbf2-42c5-8055-117368c75e1c","Type":"ContainerDied","Data":"efe3ee44831644dd74d19872caf60f9c9bb8a05f7272cf4dab15e7861077bc18"}
Sep 16 12:25:14.611754 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:14.611747    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628"
Sep 16 12:25:14.612199 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:14.611794    2579 scope.go:117] "RemoveContainer" containerID="9bbaaa3abcaea59f77baaebb50acd3a74644f571226166e45a62db3ee4cf9819"
Sep 16 12:25:14.636235 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:14.636202    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628"]
Sep 16 12:25:14.639898 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:14.639867    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cadab1d8-6fb59c7bc4-6z628"]
Sep 16 12:25:14.783531 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:14.783489    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a51e67b6-cbf2-42c5-8055-117368c75e1c" path="/var/lib/kubelet/pods/a51e67b6-cbf2-42c5-8055-117368c75e1c/volumes"
Sep 16 12:25:23.333827 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:23.333773    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb"]
Sep 16 12:25:23.334305 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:23.334059    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb" podUID="aa760fc8-008d-40f6-b924-021e43a69199" containerName="maas-api" containerID="cri-o://66f28b555b55bf8f81824b7285d30da5672dffd46e84a838528b830ce9feb957" gracePeriod=30
Sep 16 12:25:23.594474 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:23.594409    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb"
Sep 16 12:25:23.627024 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:23.626210    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa760fc8-008d-40f6-b924-021e43a69199-maas-api-tls\") pod \"aa760fc8-008d-40f6-b924-021e43a69199\" (UID: \"aa760fc8-008d-40f6-b924-021e43a69199\") "
Sep 16 12:25:23.627024 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:23.626266    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa760fc8-008d-40f6-b924-021e43a69199-metrics-certs\") pod \"aa760fc8-008d-40f6-b924-021e43a69199\" (UID: \"aa760fc8-008d-40f6-b924-021e43a69199\") "
Sep 16 12:25:23.627024 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:23.626338    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa760fc8-008d-40f6-b924-021e43a69199-openshift-service-ca\") pod \"aa760fc8-008d-40f6-b924-021e43a69199\" (UID: \"aa760fc8-008d-40f6-b924-021e43a69199\") "
Sep 16 12:25:23.627024 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:23.626465    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pwdq\" (UniqueName: \"kubernetes.io/projected/aa760fc8-008d-40f6-b924-021e43a69199-kube-api-access-7pwdq\") pod \"aa760fc8-008d-40f6-b924-021e43a69199\" (UID: \"aa760fc8-008d-40f6-b924-021e43a69199\") "
Sep 16 12:25:23.630947 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:23.630897    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa760fc8-008d-40f6-b924-021e43a69199-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aa760fc8-008d-40f6-b924-021e43a69199" (UID: "aa760fc8-008d-40f6-b924-021e43a69199"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:25:23.631356 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:23.631207    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa760fc8-008d-40f6-b924-021e43a69199-openshift-service-ca\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:25:23.631356 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:23.631284    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa760fc8-008d-40f6-b924-021e43a69199-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aa760fc8-008d-40f6-b924-021e43a69199" (UID: "aa760fc8-008d-40f6-b924-021e43a69199"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:25:23.633948 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:23.633907    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa760fc8-008d-40f6-b924-021e43a69199-kube-api-access-7pwdq" (OuterVolumeSpecName: "kube-api-access-7pwdq") pod "aa760fc8-008d-40f6-b924-021e43a69199" (UID: "aa760fc8-008d-40f6-b924-021e43a69199"). InnerVolumeSpecName "kube-api-access-7pwdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:25:23.658358 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:23.650813    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa760fc8-008d-40f6-b924-021e43a69199-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "aa760fc8-008d-40f6-b924-021e43a69199" (UID: "aa760fc8-008d-40f6-b924-021e43a69199"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:25:23.675757 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:23.675723    2579 generic.go:356] "Generic (PLEG): container finished" podID="aa760fc8-008d-40f6-b924-021e43a69199" containerID="66f28b555b55bf8f81824b7285d30da5672dffd46e84a838528b830ce9feb957" exitCode=0
Sep 16 12:25:23.675937 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:23.675788    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb"
Sep 16 12:25:23.675937 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:23.675804    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb" event={"ID":"aa760fc8-008d-40f6-b924-021e43a69199","Type":"ContainerDied","Data":"66f28b555b55bf8f81824b7285d30da5672dffd46e84a838528b830ce9feb957"}
Sep 16 12:25:23.675937 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:23.675865    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb" event={"ID":"aa760fc8-008d-40f6-b924-021e43a69199","Type":"ContainerDied","Data":"0bb1af5aac91363df7a4b20c48648a8fd7f54101e6d9f6f308236e7714225dee"}
Sep 16 12:25:23.675937 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:23.675899    2579 scope.go:117] "RemoveContainer" containerID="66f28b555b55bf8f81824b7285d30da5672dffd46e84a838528b830ce9feb957"
Sep 16 12:25:23.688519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:23.688483    2579 scope.go:117] "RemoveContainer" containerID="66f28b555b55bf8f81824b7285d30da5672dffd46e84a838528b830ce9feb957"
Sep 16 12:25:23.688896 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:25:23.688873    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66f28b555b55bf8f81824b7285d30da5672dffd46e84a838528b830ce9feb957\": container with ID starting with 66f28b555b55bf8f81824b7285d30da5672dffd46e84a838528b830ce9feb957 not found: ID does not exist" containerID="66f28b555b55bf8f81824b7285d30da5672dffd46e84a838528b830ce9feb957"
Sep 16 12:25:23.689001 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:23.688906    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66f28b555b55bf8f81824b7285d30da5672dffd46e84a838528b830ce9feb957"} err="failed to get container status \"66f28b555b55bf8f81824b7285d30da5672dffd46e84a838528b830ce9feb957\": rpc error: code = NotFound desc = could not find container \"66f28b555b55bf8f81824b7285d30da5672dffd46e84a838528b830ce9feb957\": container with ID starting with 66f28b555b55bf8f81824b7285d30da5672dffd46e84a838528b830ce9feb957 not found: ID does not exist"
Sep 16 12:25:23.701750 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:23.701727    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb"]
Sep 16 12:25:23.707029 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:23.707006    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-f1730c-7bb7d69b95-f5rsb"]
Sep 16 12:25:23.732163 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:23.732119    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pwdq\" (UniqueName: \"kubernetes.io/projected/aa760fc8-008d-40f6-b924-021e43a69199-kube-api-access-7pwdq\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:25:23.732163 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:23.732159    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa760fc8-008d-40f6-b924-021e43a69199-maas-api-tls\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:25:23.732406 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:23.732174    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa760fc8-008d-40f6-b924-021e43a69199-metrics-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:25:24.780534 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:24.780503    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa760fc8-008d-40f6-b924-021e43a69199" path="/var/lib/kubelet/pods/aa760fc8-008d-40f6-b924-021e43a69199/volumes"
Sep 16 12:25:30.021216 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:30.021184    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"]
Sep 16 12:25:30.021712 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:30.021457    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg" podUID="756d5144-227c-42ad-948f-72abbf8c8fa0" containerName="istio-proxy" containerID="cri-o://cc14db5de54d7058e83957b32b1ae744fc122fb2d222f9158e6f06e767ee9131" gracePeriod=30
Sep 16 12:25:34.159017 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:34.158976    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:25:35.280427 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.280400    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:25:35.345322 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.345296    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/756d5144-227c-42ad-948f-72abbf8c8fa0-istio-token\") pod \"756d5144-227c-42ad-948f-72abbf8c8fa0\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") "
Sep 16 12:25:35.345470 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.345363    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-workload-certs\") pod \"756d5144-227c-42ad-948f-72abbf8c8fa0\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") "
Sep 16 12:25:35.345470 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.345390    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rlvd\" (UniqueName: \"kubernetes.io/projected/756d5144-227c-42ad-948f-72abbf8c8fa0-kube-api-access-5rlvd\") pod \"756d5144-227c-42ad-948f-72abbf8c8fa0\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") "
Sep 16 12:25:35.345470 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.345448    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-istio-envoy\") pod \"756d5144-227c-42ad-948f-72abbf8c8fa0\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") "
Sep 16 12:25:35.345593 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.345491    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-credential-socket\") pod \"756d5144-227c-42ad-948f-72abbf8c8fa0\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") "
Sep 16 12:25:35.345593 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.345528    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/756d5144-227c-42ad-948f-72abbf8c8fa0-istio-podinfo\") pod \"756d5144-227c-42ad-948f-72abbf8c8fa0\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") "
Sep 16 12:25:35.345593 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.345565    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/756d5144-227c-42ad-948f-72abbf8c8fa0-istiod-ca-cert\") pod \"756d5144-227c-42ad-948f-72abbf8c8fa0\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") "
Sep 16 12:25:35.345785 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.345595    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-workload-socket\") pod \"756d5144-227c-42ad-948f-72abbf8c8fa0\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") "
Sep 16 12:25:35.345785 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.345624    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-istio-data\") pod \"756d5144-227c-42ad-948f-72abbf8c8fa0\" (UID: \"756d5144-227c-42ad-948f-72abbf8c8fa0\") "
Sep 16 12:25:35.345894 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.345841    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "756d5144-227c-42ad-948f-72abbf8c8fa0" (UID: "756d5144-227c-42ad-948f-72abbf8c8fa0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:35.346094 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.345998    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/756d5144-227c-42ad-948f-72abbf8c8fa0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "756d5144-227c-42ad-948f-72abbf8c8fa0" (UID: "756d5144-227c-42ad-948f-72abbf8c8fa0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:25:35.346094 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.346036    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "756d5144-227c-42ad-948f-72abbf8c8fa0" (UID: "756d5144-227c-42ad-948f-72abbf8c8fa0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:35.346237 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.346112    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-credential-socket\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:25:35.346295 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.346252    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-istio-data" (OuterVolumeSpecName: "istio-data") pod "756d5144-227c-42ad-948f-72abbf8c8fa0" (UID: "756d5144-227c-42ad-948f-72abbf8c8fa0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:35.346349 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.346312    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "756d5144-227c-42ad-948f-72abbf8c8fa0" (UID: "756d5144-227c-42ad-948f-72abbf8c8fa0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:35.347688 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.347644    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "756d5144-227c-42ad-948f-72abbf8c8fa0" (UID: "756d5144-227c-42ad-948f-72abbf8c8fa0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:25:35.347795 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.347699    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/756d5144-227c-42ad-948f-72abbf8c8fa0-kube-api-access-5rlvd" (OuterVolumeSpecName: "kube-api-access-5rlvd") pod "756d5144-227c-42ad-948f-72abbf8c8fa0" (UID: "756d5144-227c-42ad-948f-72abbf8c8fa0"). InnerVolumeSpecName "kube-api-access-5rlvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:25:35.347795 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.347726    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/756d5144-227c-42ad-948f-72abbf8c8fa0-istio-token" (OuterVolumeSpecName: "istio-token") pod "756d5144-227c-42ad-948f-72abbf8c8fa0" (UID: "756d5144-227c-42ad-948f-72abbf8c8fa0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:25:35.347872 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.347805    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/756d5144-227c-42ad-948f-72abbf8c8fa0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "756d5144-227c-42ad-948f-72abbf8c8fa0" (UID: "756d5144-227c-42ad-948f-72abbf8c8fa0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:25:35.447120 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.447081    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/756d5144-227c-42ad-948f-72abbf8c8fa0-istio-podinfo\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:25:35.447120 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.447114    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/756d5144-227c-42ad-948f-72abbf8c8fa0-istiod-ca-cert\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:25:35.447120 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.447123    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-workload-socket\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:25:35.447120 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.447131    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-istio-data\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:25:35.447380 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.447140    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/756d5144-227c-42ad-948f-72abbf8c8fa0-istio-token\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:25:35.447380 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.447147    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-workload-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:25:35.447380 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.447155    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rlvd\" (UniqueName: \"kubernetes.io/projected/756d5144-227c-42ad-948f-72abbf8c8fa0-kube-api-access-5rlvd\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:25:35.447380 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.447163    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/756d5144-227c-42ad-948f-72abbf8c8fa0-istio-envoy\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:25:35.748779 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.748697    2579 generic.go:356] "Generic (PLEG): container finished" podID="756d5144-227c-42ad-948f-72abbf8c8fa0" containerID="cc14db5de54d7058e83957b32b1ae744fc122fb2d222f9158e6f06e767ee9131" exitCode=0
Sep 16 12:25:35.748779 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.748746    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"
Sep 16 12:25:35.748966 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.748781    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg" event={"ID":"756d5144-227c-42ad-948f-72abbf8c8fa0","Type":"ContainerDied","Data":"cc14db5de54d7058e83957b32b1ae744fc122fb2d222f9158e6f06e767ee9131"}
Sep 16 12:25:35.748966 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.748837    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg" event={"ID":"756d5144-227c-42ad-948f-72abbf8c8fa0","Type":"ContainerDied","Data":"7d4cd71f4a9bd68bccc0e0a9b134967f2fe2c57837732483fd66070b787cb330"}
Sep 16 12:25:35.748966 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.748859    2579 scope.go:117] "RemoveContainer" containerID="cc14db5de54d7058e83957b32b1ae744fc122fb2d222f9158e6f06e767ee9131"
Sep 16 12:25:35.758522 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.758503    2579 scope.go:117] "RemoveContainer" containerID="4ea024ea239d155437d088b955e4cb1f5ff36569ce85b4b2891f3eac7964d4d8"
Sep 16 12:25:35.767418 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.767400    2579 scope.go:117] "RemoveContainer" containerID="cc14db5de54d7058e83957b32b1ae744fc122fb2d222f9158e6f06e767ee9131"
Sep 16 12:25:35.767686 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:25:35.767647    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc14db5de54d7058e83957b32b1ae744fc122fb2d222f9158e6f06e767ee9131\": container with ID starting with cc14db5de54d7058e83957b32b1ae744fc122fb2d222f9158e6f06e767ee9131 not found: ID does not exist" containerID="cc14db5de54d7058e83957b32b1ae744fc122fb2d222f9158e6f06e767ee9131"
Sep 16 12:25:35.767768 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.767696    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc14db5de54d7058e83957b32b1ae744fc122fb2d222f9158e6f06e767ee9131"} err="failed to get container status \"cc14db5de54d7058e83957b32b1ae744fc122fb2d222f9158e6f06e767ee9131\": rpc error: code = NotFound desc = could not find container \"cc14db5de54d7058e83957b32b1ae744fc122fb2d222f9158e6f06e767ee9131\": container with ID starting with cc14db5de54d7058e83957b32b1ae744fc122fb2d222f9158e6f06e767ee9131 not found: ID does not exist"
Sep 16 12:25:35.767768 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.767717    2579 scope.go:117] "RemoveContainer" containerID="4ea024ea239d155437d088b955e4cb1f5ff36569ce85b4b2891f3eac7964d4d8"
Sep 16 12:25:35.768005 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:25:35.767983    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ea024ea239d155437d088b955e4cb1f5ff36569ce85b4b2891f3eac7964d4d8\": container with ID starting with 4ea024ea239d155437d088b955e4cb1f5ff36569ce85b4b2891f3eac7964d4d8 not found: ID does not exist" containerID="4ea024ea239d155437d088b955e4cb1f5ff36569ce85b4b2891f3eac7964d4d8"
Sep 16 12:25:35.768062 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.768013    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea024ea239d155437d088b955e4cb1f5ff36569ce85b4b2891f3eac7964d4d8"} err="failed to get container status \"4ea024ea239d155437d088b955e4cb1f5ff36569ce85b4b2891f3eac7964d4d8\": rpc error: code = NotFound desc = could not find container \"4ea024ea239d155437d088b955e4cb1f5ff36569ce85b4b2891f3eac7964d4d8\": container with ID starting with 4ea024ea239d155437d088b955e4cb1f5ff36569ce85b4b2891f3eac7964d4d8 not found: ID does not exist"
Sep 16 12:25:35.771911 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.771890    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"]
Sep 16 12:25:35.776531 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:35.776510    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-f1730c-openshift-default-8465fb68f6-452zg"]
Sep 16 12:25:36.782334 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:36.782299    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="756d5144-227c-42ad-948f-72abbf8c8fa0" path="/var/lib/kubelet/pods/756d5144-227c-42ad-948f-72abbf8c8fa0/volumes"
Sep 16 12:25:41.105506 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.105472    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl"]
Sep 16 12:25:41.105967 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.105886    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="756d5144-227c-42ad-948f-72abbf8c8fa0" containerName="istio-proxy"
Sep 16 12:25:41.105967 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.105898    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="756d5144-227c-42ad-948f-72abbf8c8fa0" containerName="istio-proxy"
Sep 16 12:25:41.105967 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.105919    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa760fc8-008d-40f6-b924-021e43a69199" containerName="maas-api"
Sep 16 12:25:41.105967 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.105925    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa760fc8-008d-40f6-b924-021e43a69199" containerName="maas-api"
Sep 16 12:25:41.105967 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.105934    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a51e67b6-cbf2-42c5-8055-117368c75e1c" containerName="maas-api"
Sep 16 12:25:41.105967 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.105941    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a51e67b6-cbf2-42c5-8055-117368c75e1c" containerName="maas-api"
Sep 16 12:25:41.105967 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.105952    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="756d5144-227c-42ad-948f-72abbf8c8fa0" containerName="istio-proxy"
Sep 16 12:25:41.105967 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.105957    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="756d5144-227c-42ad-948f-72abbf8c8fa0" containerName="istio-proxy"
Sep 16 12:25:41.106215 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.106037    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="756d5144-227c-42ad-948f-72abbf8c8fa0" containerName="istio-proxy"
Sep 16 12:25:41.106215 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.106046    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a51e67b6-cbf2-42c5-8055-117368c75e1c" containerName="maas-api"
Sep 16 12:25:41.106215 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.106054    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="756d5144-227c-42ad-948f-72abbf8c8fa0" containerName="istio-proxy"
Sep 16 12:25:41.106215 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.106062    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa760fc8-008d-40f6-b924-021e43a69199" containerName="maas-api"
Sep 16 12:25:41.110891 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.110861    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl"
Sep 16 12:25:41.113511 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.113482    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-cadd00-serving-cert\""
Sep 16 12:25:41.113719 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.113493    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 12:25:41.113828 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.113790    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 12:25:41.114622 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.114599    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4gkmq\""
Sep 16 12:25:41.114752 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.114603    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:25:41.119248 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.119185    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl"]
Sep 16 12:25:41.205242 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.205199    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c7be173-8ad3-42ce-ab28-c5f73f349438-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl\" (UID: \"5c7be173-8ad3-42ce-ab28-c5f73f349438\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl"
Sep 16 12:25:41.205421 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.205260    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c7be173-8ad3-42ce-ab28-c5f73f349438-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl\" (UID: \"5c7be173-8ad3-42ce-ab28-c5f73f349438\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl"
Sep 16 12:25:41.205421 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.205289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7be173-8ad3-42ce-ab28-c5f73f349438-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl\" (UID: \"5c7be173-8ad3-42ce-ab28-c5f73f349438\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl"
Sep 16 12:25:41.205421 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.205369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z829\" (UniqueName: \"kubernetes.io/projected/5c7be173-8ad3-42ce-ab28-c5f73f349438-kube-api-access-4z829\") pod \"maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl\" (UID: \"5c7be173-8ad3-42ce-ab28-c5f73f349438\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl"
Sep 16 12:25:41.241639 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.241602    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"]
Sep 16 12:25:41.245860 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.245842    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:25:41.248556 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.248535    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-52c0e5\"/\"kube-root-ca.crt\""
Sep 16 12:25:41.248843 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.248645    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-52c0e5\"/\"openshift-service-ca.crt\""
Sep 16 12:25:41.248843 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.248702    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-52c0e5\"/\"noten-model-52c0e5-kserve-self-signed-certs\""
Sep 16 12:25:41.249029 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.248720    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-52c0e5\"/\"default-dockercfg-7xq5t\""
Sep 16 12:25:41.250875 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.250853    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"]
Sep 16 12:25:41.306595 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.306560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z829\" (UniqueName: \"kubernetes.io/projected/5c7be173-8ad3-42ce-ab28-c5f73f349438-kube-api-access-4z829\") pod \"maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl\" (UID: \"5c7be173-8ad3-42ce-ab28-c5f73f349438\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl"
Sep 16 12:25:41.306821 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.306629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c7be173-8ad3-42ce-ab28-c5f73f349438-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl\" (UID: \"5c7be173-8ad3-42ce-ab28-c5f73f349438\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl"
Sep 16 12:25:41.306821 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.306712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c7be173-8ad3-42ce-ab28-c5f73f349438-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl\" (UID: \"5c7be173-8ad3-42ce-ab28-c5f73f349438\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl"
Sep 16 12:25:41.306821 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.306743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7be173-8ad3-42ce-ab28-c5f73f349438-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl\" (UID: \"5c7be173-8ad3-42ce-ab28-c5f73f349438\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl"
Sep 16 12:25:41.307048 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:25:41.306886    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-serving-cert: secret "maas-api-e2e-shared-a-w2-cadd00-serving-cert" not found
Sep 16 12:25:41.307048 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:25:41.306955    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c7be173-8ad3-42ce-ab28-c5f73f349438-maas-api-tls podName:5c7be173-8ad3-42ce-ab28-c5f73f349438 nodeName:}" failed. No retries permitted until 2026-09-16 12:25:41.806936449 +0000 UTC m=+1335.508007762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c7be173-8ad3-42ce-ab28-c5f73f349438-maas-api-tls") pod "maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl" (UID: "5c7be173-8ad3-42ce-ab28-c5f73f349438") : secret "maas-api-e2e-shared-a-w2-cadd00-serving-cert" not found
Sep 16 12:25:41.307259 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.307240    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c7be173-8ad3-42ce-ab28-c5f73f349438-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl\" (UID: \"5c7be173-8ad3-42ce-ab28-c5f73f349438\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl"
Sep 16 12:25:41.309160 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.309130    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7be173-8ad3-42ce-ab28-c5f73f349438-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl\" (UID: \"5c7be173-8ad3-42ce-ab28-c5f73f349438\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl"
Sep 16 12:25:41.316962 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.316935    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z829\" (UniqueName: \"kubernetes.io/projected/5c7be173-8ad3-42ce-ab28-c5f73f349438-kube-api-access-4z829\") pod \"maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl\" (UID: \"5c7be173-8ad3-42ce-ab28-c5f73f349438\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl"
Sep 16 12:25:41.408186 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.407821    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h87dd\" (UniqueName: \"kubernetes.io/projected/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-kube-api-access-h87dd\") pod \"noten-model-52c0e5-kserve-65d78d87cf-pdxrr\" (UID: \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\") " pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:25:41.408186 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.407895    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-tmp-dir\") pod \"noten-model-52c0e5-kserve-65d78d87cf-pdxrr\" (UID: \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\") " pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:25:41.408186 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.407966    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-tls-certs\") pod \"noten-model-52c0e5-kserve-65d78d87cf-pdxrr\" (UID: \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\") " pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:25:41.408186 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.408035    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-home\") pod \"noten-model-52c0e5-kserve-65d78d87cf-pdxrr\" (UID: \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\") " pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:25:41.408542 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.408179    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-model-cache\") pod \"noten-model-52c0e5-kserve-65d78d87cf-pdxrr\" (UID: \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\") " pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:25:41.408542 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.408221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-dshm\") pod \"noten-model-52c0e5-kserve-65d78d87cf-pdxrr\" (UID: \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\") " pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:25:41.509181 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.509138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-model-cache\") pod \"noten-model-52c0e5-kserve-65d78d87cf-pdxrr\" (UID: \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\") " pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:25:41.509181 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.509187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-dshm\") pod \"noten-model-52c0e5-kserve-65d78d87cf-pdxrr\" (UID: \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\") " pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:25:41.509444 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.509250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h87dd\" (UniqueName: \"kubernetes.io/projected/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-kube-api-access-h87dd\") pod \"noten-model-52c0e5-kserve-65d78d87cf-pdxrr\" (UID: \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\") " pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:25:41.509444 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.509269    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-tmp-dir\") pod \"noten-model-52c0e5-kserve-65d78d87cf-pdxrr\" (UID: \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\") " pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:25:41.509444 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.509313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-tls-certs\") pod \"noten-model-52c0e5-kserve-65d78d87cf-pdxrr\" (UID: \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\") " pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:25:41.509444 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.509354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-home\") pod \"noten-model-52c0e5-kserve-65d78d87cf-pdxrr\" (UID: \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\") " pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:25:41.509696 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.509625    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-model-cache\") pod \"noten-model-52c0e5-kserve-65d78d87cf-pdxrr\" (UID: \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\") " pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:25:41.509755 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.509722    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-tmp-dir\") pod \"noten-model-52c0e5-kserve-65d78d87cf-pdxrr\" (UID: \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\") " pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:25:41.509791 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.509756    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-home\") pod \"noten-model-52c0e5-kserve-65d78d87cf-pdxrr\" (UID: \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\") " pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:25:41.511449 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.511428    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-dshm\") pod \"noten-model-52c0e5-kserve-65d78d87cf-pdxrr\" (UID: \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\") " pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:25:41.511710 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.511693    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-tls-certs\") pod \"noten-model-52c0e5-kserve-65d78d87cf-pdxrr\" (UID: \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\") " pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:25:41.519640 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.519616    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h87dd\" (UniqueName: \"kubernetes.io/projected/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-kube-api-access-h87dd\") pod \"noten-model-52c0e5-kserve-65d78d87cf-pdxrr\" (UID: \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\") " pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:25:41.557220 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.557192    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:25:41.812169 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.812087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c7be173-8ad3-42ce-ab28-c5f73f349438-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl\" (UID: \"5c7be173-8ad3-42ce-ab28-c5f73f349438\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl"
Sep 16 12:25:41.814954 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.814926    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c7be173-8ad3-42ce-ab28-c5f73f349438-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl\" (UID: \"5c7be173-8ad3-42ce-ab28-c5f73f349438\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl"
Sep 16 12:25:41.840020 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:41.839993    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"]
Sep 16 12:25:41.841368 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:25:41.841340    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d9ef162_a4e2_4cce_9e8b_d7e1e58f4cbe.slice/crio-950def2dde973915668acb2ad6e60a8482ce79411c4a1293590b2113fda854b2 WatchSource:0}: Error finding container 950def2dde973915668acb2ad6e60a8482ce79411c4a1293590b2113fda854b2: Status 404 returned error can't find the container with id 950def2dde973915668acb2ad6e60a8482ce79411c4a1293590b2113fda854b2
Sep 16 12:25:42.024260 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:42.024222    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl"
Sep 16 12:25:42.303516 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:42.303491    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl"]
Sep 16 12:25:42.304822 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:25:42.304800    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7be173_8ad3_42ce_ab28_c5f73f349438.slice/crio-7fa4301c2954625b8b309fba2ae226358a3e5028b581aaa05e44e850d6243193 WatchSource:0}: Error finding container 7fa4301c2954625b8b309fba2ae226358a3e5028b581aaa05e44e850d6243193: Status 404 returned error can't find the container with id 7fa4301c2954625b8b309fba2ae226358a3e5028b581aaa05e44e850d6243193
Sep 16 12:25:42.306339 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:42.306318    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:25:42.795201 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:42.795160    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl" event={"ID":"5c7be173-8ad3-42ce-ab28-c5f73f349438","Type":"ContainerStarted","Data":"bcc271a8180652366dadba181e14f89452a7a7405f296db8babaece027ac53bf"}
Sep 16 12:25:42.795395 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:42.795232    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl" event={"ID":"5c7be173-8ad3-42ce-ab28-c5f73f349438","Type":"ContainerStarted","Data":"7fa4301c2954625b8b309fba2ae226358a3e5028b581aaa05e44e850d6243193"}
Sep 16 12:25:42.795395 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:42.795268    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl"
Sep 16 12:25:42.796737 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:42.796713    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr" event={"ID":"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe","Type":"ContainerStarted","Data":"279ece0c1bce192c87945e446bbfae932d6dba13f6faf1859395b27686945919"}
Sep 16 12:25:42.796737 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:42.796744    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr" event={"ID":"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe","Type":"ContainerStarted","Data":"950def2dde973915668acb2ad6e60a8482ce79411c4a1293590b2113fda854b2"}
Sep 16 12:25:42.816270 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:42.816219    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl" podStartSLOduration=1.442937712 podStartE2EDuration="1.816200623s" totalImagesPullingTime="373.262911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:25:42.812865879 +0000 UTC m=+1336.513937206" watchObservedRunningTime="2026-09-16 12:25:42.816200623 +0000 UTC m=+1336.517271979"
Sep 16 12:25:42.832279 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:42.832225    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr" podStartSLOduration=1.8322090800000002 podStartE2EDuration="1.83220908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:25:42.829818558 +0000 UTC m=+1336.530889884" watchObservedRunningTime="2026-09-16 12:25:42.83220908 +0000 UTC m=+1336.533280406"
Sep 16 12:25:48.808984 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:48.808954    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl"
Sep 16 12:25:51.557743 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:51.557688    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:25:51.558230 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:51.557784    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:25:51.570827 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:51.570805    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:25:51.860078 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:51.860046    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:25:57.622135 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:25:57.622100    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"]
Sep 16 12:26:04.878585 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:04.878542    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:26:08.202049 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.202015    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"]
Sep 16 12:26:08.205035 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.205013    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.208347 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.208325    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-51ae50-openshift-default-dockercfg-phrqt\""
Sep 16 12:26:08.220568 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.220535    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"]
Sep 16 12:26:08.274348 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.274309    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b4eb96-7651-48ab-9256-3614eeb10f6c-istiod-ca-cert\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.274519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.274374    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-workload-socket\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.274519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.274401    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-istio-data\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.274519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.274443    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b4eb96-7651-48ab-9256-3614eeb10f6c-istio-podinfo\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.274519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.274468    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-workload-certs\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.274519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.274517    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-credential-socket\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.274736 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.274578    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-istio-envoy\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.274736 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.274606    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b4eb96-7651-48ab-9256-3614eeb10f6c-istio-token\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.274736 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.274625    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7ggl\" (UniqueName: \"kubernetes.io/projected/33b4eb96-7651-48ab-9256-3614eeb10f6c-kube-api-access-s7ggl\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.375478 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.375440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-workload-socket\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.375478 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.375479    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-istio-data\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.375753 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.375529    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b4eb96-7651-48ab-9256-3614eeb10f6c-istio-podinfo\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.375753 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.375549    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-workload-certs\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.375753 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.375588    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-credential-socket\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.375753 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.375614    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-istio-envoy\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.375753 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.375632    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b4eb96-7651-48ab-9256-3614eeb10f6c-istio-token\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.376021 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.375750    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7ggl\" (UniqueName: \"kubernetes.io/projected/33b4eb96-7651-48ab-9256-3614eeb10f6c-kube-api-access-s7ggl\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.376021 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.375829    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b4eb96-7651-48ab-9256-3614eeb10f6c-istiod-ca-cert\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.376021 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.375907    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-workload-socket\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.376452 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.376427    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-workload-certs\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.376583 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.376468    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-credential-socket\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.376583 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.376487    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-istio-data\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.376583 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.376545    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b4eb96-7651-48ab-9256-3614eeb10f6c-istiod-ca-cert\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.378269 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.378234    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b4eb96-7651-48ab-9256-3614eeb10f6c-istio-podinfo\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.378403 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.378319    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-istio-envoy\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.392869 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.392845    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b4eb96-7651-48ab-9256-3614eeb10f6c-istio-token\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.393017 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.393001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7ggl\" (UniqueName: \"kubernetes.io/projected/33b4eb96-7651-48ab-9256-3614eeb10f6c-kube-api-access-s7ggl\") pod \"e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") " pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.515585 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.515488    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:08.793402 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.793094    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"]
Sep 16 12:26:08.796397 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:26:08.796366    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b4eb96_7651_48ab_9256_3614eeb10f6c.slice/crio-0c03794086ec082fc4d7e313f342e4c82437e865a425d2297ce99992529fc4da WatchSource:0}: Error finding container 0c03794086ec082fc4d7e313f342e4c82437e865a425d2297ce99992529fc4da: Status 404 returned error can't find the container with id 0c03794086ec082fc4d7e313f342e4c82437e865a425d2297ce99992529fc4da
Sep 16 12:26:08.798355 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.798324    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:26:08.798412 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.798390    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:26:08.798448 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.798416    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:26:08.950277 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.950244    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7" event={"ID":"33b4eb96-7651-48ab-9256-3614eeb10f6c","Type":"ContainerStarted","Data":"c7f6d20ddf4275b3aa4a9a725da93409fbe273ae93cd796439740aea9be792ed"}
Sep 16 12:26:08.950447 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.950306    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7" event={"ID":"33b4eb96-7651-48ab-9256-3614eeb10f6c","Type":"ContainerStarted","Data":"0c03794086ec082fc4d7e313f342e4c82437e865a425d2297ce99992529fc4da"}
Sep 16 12:26:08.990462 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:08.990399    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7" podStartSLOduration=0.990378644 podStartE2EDuration="990.378644ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:26:08.98580971 +0000 UTC m=+1362.686881036" watchObservedRunningTime="2026-09-16 12:26:08.990378644 +0000 UTC m=+1362.691449972"
Sep 16 12:26:09.515980 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:09.515946    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:09.521129 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:09.521106    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:09.955722 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:09.955693    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:09.956744 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:09.956723    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:12.641868 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:12.641807    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr" podUID="6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe" containerName="main" containerID="cri-o://279ece0c1bce192c87945e446bbfae932d6dba13f6faf1859395b27686945919" gracePeriod=45
Sep 16 12:26:12.901885 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:12.901818    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:26:12.974646 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:12.974609    2579 generic.go:356] "Generic (PLEG): container finished" podID="6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe" containerID="279ece0c1bce192c87945e446bbfae932d6dba13f6faf1859395b27686945919" exitCode=0
Sep 16 12:26:12.974845 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:12.974665    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"
Sep 16 12:26:12.974845 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:12.974693    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr" event={"ID":"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe","Type":"ContainerDied","Data":"279ece0c1bce192c87945e446bbfae932d6dba13f6faf1859395b27686945919"}
Sep 16 12:26:12.974845 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:12.974743    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr" event={"ID":"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe","Type":"ContainerDied","Data":"950def2dde973915668acb2ad6e60a8482ce79411c4a1293590b2113fda854b2"}
Sep 16 12:26:12.974845 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:12.974761    2579 scope.go:117] "RemoveContainer" containerID="279ece0c1bce192c87945e446bbfae932d6dba13f6faf1859395b27686945919"
Sep 16 12:26:12.984176 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:12.984152    2579 scope.go:117] "RemoveContainer" containerID="279ece0c1bce192c87945e446bbfae932d6dba13f6faf1859395b27686945919"
Sep 16 12:26:12.984485 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:26:12.984467    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"279ece0c1bce192c87945e446bbfae932d6dba13f6faf1859395b27686945919\": container with ID starting with 279ece0c1bce192c87945e446bbfae932d6dba13f6faf1859395b27686945919 not found: ID does not exist" containerID="279ece0c1bce192c87945e446bbfae932d6dba13f6faf1859395b27686945919"
Sep 16 12:26:12.984558 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:12.984494    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"279ece0c1bce192c87945e446bbfae932d6dba13f6faf1859395b27686945919"} err="failed to get container status \"279ece0c1bce192c87945e446bbfae932d6dba13f6faf1859395b27686945919\": rpc error: code = NotFound desc = could not find container \"279ece0c1bce192c87945e446bbfae932d6dba13f6faf1859395b27686945919\": container with ID starting with 279ece0c1bce192c87945e446bbfae932d6dba13f6faf1859395b27686945919 not found: ID does not exist"
Sep 16 12:26:13.035022 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:13.034985    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-tls-certs\") pod \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\" (UID: \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\") "
Sep 16 12:26:13.035022 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:13.035026    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-model-cache\") pod \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\" (UID: \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\") "
Sep 16 12:26:13.035228 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:13.035062    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-tmp-dir\") pod \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\" (UID: \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\") "
Sep 16 12:26:13.035228 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:13.035134    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-dshm\") pod \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\" (UID: \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\") "
Sep 16 12:26:13.035228 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:13.035179    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-home\") pod \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\" (UID: \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\") "
Sep 16 12:26:13.035382 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:13.035239    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h87dd\" (UniqueName: \"kubernetes.io/projected/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-kube-api-access-h87dd\") pod \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\" (UID: \"6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe\") "
Sep 16 12:26:13.035382 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:13.035334    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-model-cache" (OuterVolumeSpecName: "model-cache") pod "6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe" (UID: "6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:26:13.035382 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:13.035356    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe" (UID: "6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:26:13.035569 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:13.035546    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-home" (OuterVolumeSpecName: "home") pod "6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe" (UID: "6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:26:13.035617 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:13.035567    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-tmp-dir\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:26:13.035617 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:13.035581    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-model-cache\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:26:13.037392 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:13.037366    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe" (UID: "6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:26:13.037487 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:13.037365    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-dshm" (OuterVolumeSpecName: "dshm") pod "6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe" (UID: "6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:26:13.037487 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:13.037371    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-kube-api-access-h87dd" (OuterVolumeSpecName: "kube-api-access-h87dd") pod "6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe" (UID: "6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe"). InnerVolumeSpecName "kube-api-access-h87dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:26:13.136899 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:13.136862    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-dshm\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:26:13.136899 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:13.136891    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-home\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:26:13.136899 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:13.136901    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h87dd\" (UniqueName: \"kubernetes.io/projected/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-kube-api-access-h87dd\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:26:13.137129 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:13.136911    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe-tls-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:26:13.299482 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:13.299452    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"]
Sep 16 12:26:13.302291 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:13.302262    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-52c0e5/noten-model-52c0e5-kserve-65d78d87cf-pdxrr"]
Sep 16 12:26:14.782055 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:14.782024    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe" path="/var/lib/kubelet/pods/6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe/volumes"
Sep 16 12:26:15.109756 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:15.109725    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4"]
Sep 16 12:26:15.110185 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:15.110165    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe" containerName="main"
Sep 16 12:26:15.110185 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:15.110184    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe" containerName="main"
Sep 16 12:26:15.110326 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:15.110266    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d9ef162-a4e2-4cce-9e8b-d7e1e58f4cbe" containerName="main"
Sep 16 12:26:15.114765 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:15.114742    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4"
Sep 16 12:26:15.117351 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:15.117329    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-51ae50-serving-cert\""
Sep 16 12:26:15.124194 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:15.124172    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4"]
Sep 16 12:26:15.260544 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:15.260509    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/787efc7d-197f-4c85-9266-7e38ffaf83db-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4\" (UID: \"787efc7d-197f-4c85-9266-7e38ffaf83db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4"
Sep 16 12:26:15.260736 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:15.260574    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwn5q\" (UniqueName: \"kubernetes.io/projected/787efc7d-197f-4c85-9266-7e38ffaf83db-kube-api-access-mwn5q\") pod \"maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4\" (UID: \"787efc7d-197f-4c85-9266-7e38ffaf83db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4"
Sep 16 12:26:15.260736 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:15.260721    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/787efc7d-197f-4c85-9266-7e38ffaf83db-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4\" (UID: \"787efc7d-197f-4c85-9266-7e38ffaf83db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4"
Sep 16 12:26:15.260868 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:15.260777    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/787efc7d-197f-4c85-9266-7e38ffaf83db-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4\" (UID: \"787efc7d-197f-4c85-9266-7e38ffaf83db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4"
Sep 16 12:26:15.361467 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:15.361360    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/787efc7d-197f-4c85-9266-7e38ffaf83db-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4\" (UID: \"787efc7d-197f-4c85-9266-7e38ffaf83db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4"
Sep 16 12:26:15.361467 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:15.361435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwn5q\" (UniqueName: \"kubernetes.io/projected/787efc7d-197f-4c85-9266-7e38ffaf83db-kube-api-access-mwn5q\") pod \"maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4\" (UID: \"787efc7d-197f-4c85-9266-7e38ffaf83db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4"
Sep 16 12:26:15.361749 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:15.361524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/787efc7d-197f-4c85-9266-7e38ffaf83db-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4\" (UID: \"787efc7d-197f-4c85-9266-7e38ffaf83db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4"
Sep 16 12:26:15.361749 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:15.361565    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/787efc7d-197f-4c85-9266-7e38ffaf83db-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4\" (UID: \"787efc7d-197f-4c85-9266-7e38ffaf83db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4"
Sep 16 12:26:15.362268 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:15.362241    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/787efc7d-197f-4c85-9266-7e38ffaf83db-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4\" (UID: \"787efc7d-197f-4c85-9266-7e38ffaf83db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4"
Sep 16 12:26:15.363868 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:15.363842    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/787efc7d-197f-4c85-9266-7e38ffaf83db-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4\" (UID: \"787efc7d-197f-4c85-9266-7e38ffaf83db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4"
Sep 16 12:26:15.363950 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:15.363937    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/787efc7d-197f-4c85-9266-7e38ffaf83db-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4\" (UID: \"787efc7d-197f-4c85-9266-7e38ffaf83db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4"
Sep 16 12:26:15.370490 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:15.370460    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwn5q\" (UniqueName: \"kubernetes.io/projected/787efc7d-197f-4c85-9266-7e38ffaf83db-kube-api-access-mwn5q\") pod \"maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4\" (UID: \"787efc7d-197f-4c85-9266-7e38ffaf83db\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4"
Sep 16 12:26:15.427253 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:15.427218    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4"
Sep 16 12:26:15.700595 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:15.700566    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4"]
Sep 16 12:26:15.701992 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:26:15.701966    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod787efc7d_197f_4c85_9266_7e38ffaf83db.slice/crio-10a24ffceb5b4a4026395237446f8de188ad09a279a1a4a67185fca44bfe6403 WatchSource:0}: Error finding container 10a24ffceb5b4a4026395237446f8de188ad09a279a1a4a67185fca44bfe6403: Status 404 returned error can't find the container with id 10a24ffceb5b4a4026395237446f8de188ad09a279a1a4a67185fca44bfe6403
Sep 16 12:26:15.996690 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:15.996601    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4" event={"ID":"787efc7d-197f-4c85-9266-7e38ffaf83db","Type":"ContainerStarted","Data":"10a24ffceb5b4a4026395237446f8de188ad09a279a1a4a67185fca44bfe6403"}
Sep 16 12:26:17.004632 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:17.004592    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4" event={"ID":"787efc7d-197f-4c85-9266-7e38ffaf83db","Type":"ContainerStarted","Data":"77f922b4a277740ddd07a84449e075ebd0a2b2918dbc3082149692dd94b1e1ce"}
Sep 16 12:26:17.005105 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:17.004810    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4"
Sep 16 12:26:17.024065 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:17.024001    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4" podStartSLOduration=1.723370599 podStartE2EDuration="2.02397678s" totalImagesPullingTime="300.606181ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:26:17.021839918 +0000 UTC m=+1370.722911291" watchObservedRunningTime="2026-09-16 12:26:17.02397678 +0000 UTC m=+1370.725048106"
Sep 16 12:26:23.016337 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:23.016307    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4"
Sep 16 12:26:34.131529 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:34.131499    2579 generic.go:356] "Generic (PLEG): container finished" podID="33b4eb96-7651-48ab-9256-3614eeb10f6c" containerID="c7f6d20ddf4275b3aa4a9a725da93409fbe273ae93cd796439740aea9be792ed" exitCode=0
Sep 16 12:26:34.131956 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:34.131562    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7" event={"ID":"33b4eb96-7651-48ab-9256-3614eeb10f6c","Type":"ContainerDied","Data":"c7f6d20ddf4275b3aa4a9a725da93409fbe273ae93cd796439740aea9be792ed"}
Sep 16 12:26:34.132003 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:34.131982    2579 scope.go:117] "RemoveContainer" containerID="c7f6d20ddf4275b3aa4a9a725da93409fbe273ae93cd796439740aea9be792ed"
Sep 16 12:26:34.133136 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:34.133103    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:26:34.133256 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:34.133198    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:26:34.133256 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:34.133250    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:26:35.139200 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:35.139167    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7" event={"ID":"33b4eb96-7651-48ab-9256-3614eeb10f6c","Type":"ContainerStarted","Data":"3acf5ac04b96cfe3d2a2fa9a328eace9aa2d8b682408e3c0833552b5ac1e6667"}
Sep 16 12:26:35.516414 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:35.516309    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:35.517706 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:35.517678    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 16 12:26:35.517824 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:35.517736    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7" podUID="33b4eb96-7651-48ab-9256-3614eeb10f6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 16 12:26:36.516187 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:36.516151    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 16 12:26:36.516576 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:36.516211    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7" podUID="33b4eb96-7651-48ab-9256-3614eeb10f6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 16 12:26:37.516554 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:37.516509    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 16 12:26:37.517059 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:37.516592    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7" podUID="33b4eb96-7651-48ab-9256-3614eeb10f6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 16 12:26:38.516800 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:38.516763    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 16 12:26:38.517301 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:38.516829    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7" podUID="33b4eb96-7651-48ab-9256-3614eeb10f6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 16 12:26:39.516311 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:39.516276    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 16 12:26:39.516481 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:39.516343    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7" podUID="33b4eb96-7651-48ab-9256-3614eeb10f6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 16 12:26:39.956100 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:39.956065    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:40.516620 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:40.516580    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 16 12:26:40.516809 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:40.516649    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7" podUID="33b4eb96-7651-48ab-9256-3614eeb10f6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 16 12:26:41.519931 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:41.519900    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:41.521003 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:41.520985    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:26:41.675507 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:41.675476    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7880-fc9825"]
Sep 16 12:26:41.678530 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:41.678514    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7880-fc9825"
Sep 16 12:26:41.681176 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:41.681148    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6btc9\""
Sep 16 12:26:41.684440 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:41.684421    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7880-fc9825"]
Sep 16 12:26:41.739244 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:41.739210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnkp4\" (UniqueName: \"kubernetes.io/projected/d741fcb3-e536-4a34-894f-df2392911954-kube-api-access-cnkp4\") pod \"test-curl-7880-fc9825\" (UID: \"d741fcb3-e536-4a34-894f-df2392911954\") " pod="opendatahub/test-curl-7880-fc9825"
Sep 16 12:26:41.840094 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:41.840067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnkp4\" (UniqueName: \"kubernetes.io/projected/d741fcb3-e536-4a34-894f-df2392911954-kube-api-access-cnkp4\") pod \"test-curl-7880-fc9825\" (UID: \"d741fcb3-e536-4a34-894f-df2392911954\") " pod="opendatahub/test-curl-7880-fc9825"
Sep 16 12:26:41.848641 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:41.848617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnkp4\" (UniqueName: \"kubernetes.io/projected/d741fcb3-e536-4a34-894f-df2392911954-kube-api-access-cnkp4\") pod \"test-curl-7880-fc9825\" (UID: \"d741fcb3-e536-4a34-894f-df2392911954\") " pod="opendatahub/test-curl-7880-fc9825"
Sep 16 12:26:41.990007 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:41.989981    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7880-fc9825"
Sep 16 12:26:42.260254 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:42.260226    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7880-fc9825"]
Sep 16 12:26:42.262447 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:26:42.262397    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd741fcb3_e536_4a34_894f_df2392911954.slice/crio-4669e57971ffef4b2f370258b728ec3df58c41f2f710e479b81a0e30ff732563 WatchSource:0}: Error finding container 4669e57971ffef4b2f370258b728ec3df58c41f2f710e479b81a0e30ff732563: Status 404 returned error can't find the container with id 4669e57971ffef4b2f370258b728ec3df58c41f2f710e479b81a0e30ff732563
Sep 16 12:26:42.662221 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:42.662186    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:26:43.199210 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:43.199177    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7880-fc9825" event={"ID":"d741fcb3-e536-4a34-894f-df2392911954","Type":"ContainerStarted","Data":"e6140b946c1db4f4dd45424822cfd4a9f66805ac9dc258fbeeb423b9b9bf297e"}
Sep 16 12:26:43.199386 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:43.199220    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7880-fc9825" event={"ID":"d741fcb3-e536-4a34-894f-df2392911954","Type":"ContainerStarted","Data":"4669e57971ffef4b2f370258b728ec3df58c41f2f710e479b81a0e30ff732563"}
Sep 16 12:26:43.215006 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:43.214957    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7880-fc9825" podStartSLOduration=2.214941101 podStartE2EDuration="2.214941101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:26:43.21172293 +0000 UTC m=+1396.912794257" watchObservedRunningTime="2026-09-16 12:26:43.214941101 +0000 UTC m=+1396.916012427"
Sep 16 12:26:43.785127 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:43.785087    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7880-fc9825"]
Sep 16 12:26:43.790281 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:43.790248    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7880-fc9825"]
Sep 16 12:26:43.912599 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:43.912545    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7880-341e30"]
Sep 16 12:26:43.913330 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:43.913303    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d741fcb3-e536-4a34-894f-df2392911954" containerName="test-curl-7880-fc9825"
Sep 16 12:26:43.913330 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:43.913330    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d741fcb3-e536-4a34-894f-df2392911954" containerName="test-curl-7880-fc9825"
Sep 16 12:26:43.913510 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:43.913488    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d741fcb3-e536-4a34-894f-df2392911954" containerName="test-curl-7880-fc9825"
Sep 16 12:26:43.916360 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:43.916333    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7880-341e30"
Sep 16 12:26:43.919834 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:43.919737    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7880-341e30"]
Sep 16 12:26:44.065815 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:44.065782    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk984\" (UniqueName: \"kubernetes.io/projected/45325b4d-ab21-46a5-a55d-336df593395b-kube-api-access-sk984\") pod \"test-curl-7880-341e30\" (UID: \"45325b4d-ab21-46a5-a55d-336df593395b\") " pod="opendatahub/test-curl-7880-341e30"
Sep 16 12:26:44.167005 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:44.166916    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk984\" (UniqueName: \"kubernetes.io/projected/45325b4d-ab21-46a5-a55d-336df593395b-kube-api-access-sk984\") pod \"test-curl-7880-341e30\" (UID: \"45325b4d-ab21-46a5-a55d-336df593395b\") " pod="opendatahub/test-curl-7880-341e30"
Sep 16 12:26:44.176095 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:44.176068    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk984\" (UniqueName: \"kubernetes.io/projected/45325b4d-ab21-46a5-a55d-336df593395b-kube-api-access-sk984\") pod \"test-curl-7880-341e30\" (UID: \"45325b4d-ab21-46a5-a55d-336df593395b\") " pod="opendatahub/test-curl-7880-341e30"
Sep 16 12:26:44.205385 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:26:44.205359    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7880-fc9825" container="test-curl-7880-fc9825"
Sep 16 12:26:44.230192 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:44.230165    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7880-341e30"
Sep 16 12:26:44.500723 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:44.499992    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7880-341e30"]
Sep 16 12:26:45.212625 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:45.212581    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7880-341e30" event={"ID":"45325b4d-ab21-46a5-a55d-336df593395b","Type":"ContainerStarted","Data":"06ad6b3f6257ec2871dc547291d0be26c4285a388b3b508fb3caec2f21124979"}
Sep 16 12:26:45.212625 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:45.212634    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7880-341e30" event={"ID":"45325b4d-ab21-46a5-a55d-336df593395b","Type":"ContainerStarted","Data":"06dac88755f8973dac5c9b7778240107f3073367c1247e6b23b6eddc4bf473a0"}
Sep 16 12:26:45.213185 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:45.212621    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7880-fc9825" podUID="d741fcb3-e536-4a34-894f-df2392911954" containerName="test-curl-7880-fc9825" containerID="cri-o://e6140b946c1db4f4dd45424822cfd4a9f66805ac9dc258fbeeb423b9b9bf297e" gracePeriod=2
Sep 16 12:26:45.228341 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:45.228295    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7880-341e30" podStartSLOduration=2.228279566 podStartE2EDuration="2.228279566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:26:45.226588371 +0000 UTC m=+1398.927659697" watchObservedRunningTime="2026-09-16 12:26:45.228279566 +0000 UTC m=+1398.929350911"
Sep 16 12:26:45.782611 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:45.782568    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7880-341e30"]
Sep 16 12:26:45.786613 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:45.786584    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7880-341e30"]
Sep 16 12:26:45.899873 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:45.899836    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7880-1e55d1"]
Sep 16 12:26:45.900312 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:45.900292    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45325b4d-ab21-46a5-a55d-336df593395b" containerName="test-curl-7880-341e30"
Sep 16 12:26:45.900312 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:45.900309    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="45325b4d-ab21-46a5-a55d-336df593395b" containerName="test-curl-7880-341e30"
Sep 16 12:26:45.900498 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:45.900392    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="45325b4d-ab21-46a5-a55d-336df593395b" containerName="test-curl-7880-341e30"
Sep 16 12:26:45.902830 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:45.902812    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7880-1e55d1"
Sep 16 12:26:45.909311 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:45.909283    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7880-1e55d1"]
Sep 16 12:26:45.984824 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:45.984787    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbjnf\" (UniqueName: \"kubernetes.io/projected/08b1cbdc-9c65-4304-b845-c54891361d2a-kube-api-access-pbjnf\") pod \"test-curl-7880-1e55d1\" (UID: \"08b1cbdc-9c65-4304-b845-c54891361d2a\") " pod="opendatahub/test-curl-7880-1e55d1"
Sep 16 12:26:46.086020 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:46.085977    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbjnf\" (UniqueName: \"kubernetes.io/projected/08b1cbdc-9c65-4304-b845-c54891361d2a-kube-api-access-pbjnf\") pod \"test-curl-7880-1e55d1\" (UID: \"08b1cbdc-9c65-4304-b845-c54891361d2a\") " pod="opendatahub/test-curl-7880-1e55d1"
Sep 16 12:26:46.094021 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:46.093996    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbjnf\" (UniqueName: \"kubernetes.io/projected/08b1cbdc-9c65-4304-b845-c54891361d2a-kube-api-access-pbjnf\") pod \"test-curl-7880-1e55d1\" (UID: \"08b1cbdc-9c65-4304-b845-c54891361d2a\") " pod="opendatahub/test-curl-7880-1e55d1"
Sep 16 12:26:46.214838 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:46.214746    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7880-1e55d1"
Sep 16 12:26:46.219246 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:26:46.219227    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7880-341e30" container="test-curl-7880-341e30"
Sep 16 12:26:46.484550 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:46.484519    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7880-1e55d1"]
Sep 16 12:26:46.485254 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:26:46.485230    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08b1cbdc_9c65_4304_b845_c54891361d2a.slice/crio-cb33111b31dc72e351b822d2d2d19d9e404a56e3f8502ea4b67387b7bf41c6e7 WatchSource:0}: Error finding container cb33111b31dc72e351b822d2d2d19d9e404a56e3f8502ea4b67387b7bf41c6e7: Status 404 returned error can't find the container with id cb33111b31dc72e351b822d2d2d19d9e404a56e3f8502ea4b67387b7bf41c6e7
Sep 16 12:26:47.227085 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:47.227047    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7880-1e55d1" event={"ID":"08b1cbdc-9c65-4304-b845-c54891361d2a","Type":"ContainerStarted","Data":"4ad67f3896fdfadf31189c85b73ac44be9284447205d9bce2a87a6e4e3cd9003"}
Sep 16 12:26:47.227449 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:47.227109    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7880-1e55d1" event={"ID":"08b1cbdc-9c65-4304-b845-c54891361d2a","Type":"ContainerStarted","Data":"cb33111b31dc72e351b822d2d2d19d9e404a56e3f8502ea4b67387b7bf41c6e7"}
Sep 16 12:26:47.227449 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:47.227268    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7880-341e30" podUID="45325b4d-ab21-46a5-a55d-336df593395b" containerName="test-curl-7880-341e30" containerID="cri-o://06ad6b3f6257ec2871dc547291d0be26c4285a388b3b508fb3caec2f21124979" gracePeriod=2
Sep 16 12:26:47.242766 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:47.242723    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7880-1e55d1" podStartSLOduration=2.242708748 podStartE2EDuration="2.242708748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:26:47.239924516 +0000 UTC m=+1400.940995842" watchObservedRunningTime="2026-09-16 12:26:47.242708748 +0000 UTC m=+1400.943780073"
Sep 16 12:26:47.392921 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:47.392898    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7880-fc9825"
Sep 16 12:26:47.500480 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:47.500396    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnkp4\" (UniqueName: \"kubernetes.io/projected/d741fcb3-e536-4a34-894f-df2392911954-kube-api-access-cnkp4\") pod \"d741fcb3-e536-4a34-894f-df2392911954\" (UID: \"d741fcb3-e536-4a34-894f-df2392911954\") "
Sep 16 12:26:47.502366 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:47.502336    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d741fcb3-e536-4a34-894f-df2392911954-kube-api-access-cnkp4" (OuterVolumeSpecName: "kube-api-access-cnkp4") pod "d741fcb3-e536-4a34-894f-df2392911954" (UID: "d741fcb3-e536-4a34-894f-df2392911954"). InnerVolumeSpecName "kube-api-access-cnkp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:26:47.601839 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:47.601805    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnkp4\" (UniqueName: \"kubernetes.io/projected/d741fcb3-e536-4a34-894f-df2392911954-kube-api-access-cnkp4\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:26:47.802840 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:47.802736    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7880-1e55d1"]
Sep 16 12:26:47.806605 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:47.806580    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7880-1e55d1"]
Sep 16 12:26:47.922150 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:47.922109    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7880-9b36a1"]
Sep 16 12:26:47.922616 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:47.922596    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b1cbdc-9c65-4304-b845-c54891361d2a" containerName="test-curl-7880-1e55d1"
Sep 16 12:26:47.922753 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:47.922618    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b1cbdc-9c65-4304-b845-c54891361d2a" containerName="test-curl-7880-1e55d1"
Sep 16 12:26:47.922753 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:47.922733    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="08b1cbdc-9c65-4304-b845-c54891361d2a" containerName="test-curl-7880-1e55d1"
Sep 16 12:26:47.924933 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:47.924912    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7880-9b36a1"
Sep 16 12:26:47.933799 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:47.933772    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7880-9b36a1"]
Sep 16 12:26:48.005610 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:48.005577    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhjbd\" (UniqueName: \"kubernetes.io/projected/5fd5535a-2560-44ff-8ff0-861a68a48a02-kube-api-access-bhjbd\") pod \"test-curl-7880-9b36a1\" (UID: \"5fd5535a-2560-44ff-8ff0-861a68a48a02\") " pod="opendatahub/test-curl-7880-9b36a1"
Sep 16 12:26:48.107198 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:48.107157    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhjbd\" (UniqueName: \"kubernetes.io/projected/5fd5535a-2560-44ff-8ff0-861a68a48a02-kube-api-access-bhjbd\") pod \"test-curl-7880-9b36a1\" (UID: \"5fd5535a-2560-44ff-8ff0-861a68a48a02\") " pod="opendatahub/test-curl-7880-9b36a1"
Sep 16 12:26:48.114809 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:48.114788    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhjbd\" (UniqueName: \"kubernetes.io/projected/5fd5535a-2560-44ff-8ff0-861a68a48a02-kube-api-access-bhjbd\") pod \"test-curl-7880-9b36a1\" (UID: \"5fd5535a-2560-44ff-8ff0-861a68a48a02\") " pod="opendatahub/test-curl-7880-9b36a1"
Sep 16 12:26:48.234093 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:48.234059    2579 generic.go:356] "Generic (PLEG): container finished" podID="d741fcb3-e536-4a34-894f-df2392911954" containerID="e6140b946c1db4f4dd45424822cfd4a9f66805ac9dc258fbeeb423b9b9bf297e" exitCode=137
Sep 16 12:26:48.234528 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:48.234109    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7880-fc9825"
Sep 16 12:26:48.234528 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:48.234165    2579 scope.go:117] "RemoveContainer" containerID="e6140b946c1db4f4dd45424822cfd4a9f66805ac9dc258fbeeb423b9b9bf297e"
Sep 16 12:26:48.234528 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:26:48.234461    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7880-1e55d1" container="test-curl-7880-1e55d1"
Sep 16 12:26:48.235417 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:48.235402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7880-9b36a1"
Sep 16 12:26:48.252367 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:48.252347    2579 scope.go:117] "RemoveContainer" containerID="e6140b946c1db4f4dd45424822cfd4a9f66805ac9dc258fbeeb423b9b9bf297e"
Sep 16 12:26:48.252802 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:26:48.252757    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6140b946c1db4f4dd45424822cfd4a9f66805ac9dc258fbeeb423b9b9bf297e\": container with ID starting with e6140b946c1db4f4dd45424822cfd4a9f66805ac9dc258fbeeb423b9b9bf297e not found: ID does not exist" containerID="e6140b946c1db4f4dd45424822cfd4a9f66805ac9dc258fbeeb423b9b9bf297e"
Sep 16 12:26:48.252884 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:48.252834    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6140b946c1db4f4dd45424822cfd4a9f66805ac9dc258fbeeb423b9b9bf297e"} err="failed to get container status \"e6140b946c1db4f4dd45424822cfd4a9f66805ac9dc258fbeeb423b9b9bf297e\": rpc error: code = NotFound desc = could not find container \"e6140b946c1db4f4dd45424822cfd4a9f66805ac9dc258fbeeb423b9b9bf297e\": container with ID starting with e6140b946c1db4f4dd45424822cfd4a9f66805ac9dc258fbeeb423b9b9bf297e not found: ID does not exist"
Sep 16 12:26:48.268602 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:26:48.268569    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7880-fc9825\" is forbidden: User \"system:node:ip-10-0-137-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-30.ec2.internal' and this object" podUID="d741fcb3-e536-4a34-894f-df2392911954" pod="opendatahub/test-curl-7880-fc9825"
Sep 16 12:26:48.514363 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:48.514335    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7880-9b36a1"]
Sep 16 12:26:48.515067 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:26:48.515045    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fd5535a_2560_44ff_8ff0_861a68a48a02.slice/crio-018b821408e6d4b5abee862a8844f299a1529dbcb831d721c3584d4ea2b067b6 WatchSource:0}: Error finding container 018b821408e6d4b5abee862a8844f299a1529dbcb831d721c3584d4ea2b067b6: Status 404 returned error can't find the container with id 018b821408e6d4b5abee862a8844f299a1529dbcb831d721c3584d4ea2b067b6
Sep 16 12:26:48.781437 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:48.781355    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d741fcb3-e536-4a34-894f-df2392911954" path="/var/lib/kubelet/pods/d741fcb3-e536-4a34-894f-df2392911954/volumes"
Sep 16 12:26:49.241723 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:49.241689    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7880-9b36a1" event={"ID":"5fd5535a-2560-44ff-8ff0-861a68a48a02","Type":"ContainerStarted","Data":"8b19420867bc411ac8d3f69ce223fe23cfe20f680a4336dd77da00adcbde2f13"}
Sep 16 12:26:49.242051 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:49.241758    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7880-9b36a1" event={"ID":"5fd5535a-2560-44ff-8ff0-861a68a48a02","Type":"ContainerStarted","Data":"018b821408e6d4b5abee862a8844f299a1529dbcb831d721c3584d4ea2b067b6"}
Sep 16 12:26:49.242920 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:49.242895    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7880-1e55d1" podUID="08b1cbdc-9c65-4304-b845-c54891361d2a" containerName="test-curl-7880-1e55d1" containerID="cri-o://4ad67f3896fdfadf31189c85b73ac44be9284447205d9bce2a87a6e4e3cd9003" gracePeriod=2
Sep 16 12:26:49.255129 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:49.254936    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7880-9b36a1" podStartSLOduration=2.254915009 podStartE2EDuration="2.254915009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:26:49.254423971 +0000 UTC m=+1402.955495298" watchObservedRunningTime="2026-09-16 12:26:49.254915009 +0000 UTC m=+1402.955986336"
Sep 16 12:26:49.371917 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:49.371890    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7880-341e30"
Sep 16 12:26:49.522505 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:49.522417    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sk984\" (UniqueName: \"kubernetes.io/projected/45325b4d-ab21-46a5-a55d-336df593395b-kube-api-access-sk984\") pod \"45325b4d-ab21-46a5-a55d-336df593395b\" (UID: \"45325b4d-ab21-46a5-a55d-336df593395b\") "
Sep 16 12:26:49.524493 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:49.524466    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45325b4d-ab21-46a5-a55d-336df593395b-kube-api-access-sk984" (OuterVolumeSpecName: "kube-api-access-sk984") pod "45325b4d-ab21-46a5-a55d-336df593395b" (UID: "45325b4d-ab21-46a5-a55d-336df593395b"). InnerVolumeSpecName "kube-api-access-sk984". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:26:49.623154 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:49.623124    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk984\" (UniqueName: \"kubernetes.io/projected/45325b4d-ab21-46a5-a55d-336df593395b-kube-api-access-sk984\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:26:49.822003 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:49.821915    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7880-9b36a1"]
Sep 16 12:26:49.824936 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:49.824908    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7880-9b36a1"]
Sep 16 12:26:50.250998 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.250965    2579 generic.go:356] "Generic (PLEG): container finished" podID="45325b4d-ab21-46a5-a55d-336df593395b" containerID="06ad6b3f6257ec2871dc547291d0be26c4285a388b3b508fb3caec2f21124979" exitCode=137
Sep 16 12:26:50.251440 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.251013    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7880-341e30"
Sep 16 12:26:50.251440 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.251072    2579 scope.go:117] "RemoveContainer" containerID="06ad6b3f6257ec2871dc547291d0be26c4285a388b3b508fb3caec2f21124979"
Sep 16 12:26:50.251440 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:26:50.251363    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7880-9b36a1" container="test-curl-7880-9b36a1"
Sep 16 12:26:50.260682 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.260638    2579 scope.go:117] "RemoveContainer" containerID="06ad6b3f6257ec2871dc547291d0be26c4285a388b3b508fb3caec2f21124979"
Sep 16 12:26:50.260963 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:26:50.260947    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06ad6b3f6257ec2871dc547291d0be26c4285a388b3b508fb3caec2f21124979\": container with ID starting with 06ad6b3f6257ec2871dc547291d0be26c4285a388b3b508fb3caec2f21124979 not found: ID does not exist" containerID="06ad6b3f6257ec2871dc547291d0be26c4285a388b3b508fb3caec2f21124979"
Sep 16 12:26:50.261015 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.260972    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ad6b3f6257ec2871dc547291d0be26c4285a388b3b508fb3caec2f21124979"} err="failed to get container status \"06ad6b3f6257ec2871dc547291d0be26c4285a388b3b508fb3caec2f21124979\": rpc error: code = NotFound desc = could not find container \"06ad6b3f6257ec2871dc547291d0be26c4285a388b3b508fb3caec2f21124979\": container with ID starting with 06ad6b3f6257ec2871dc547291d0be26c4285a388b3b508fb3caec2f21124979 not found: ID does not exist"
Sep 16 12:26:50.261357 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:26:50.261338    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7880-341e30\" is forbidden: User \"system:node:ip-10-0-137-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-30.ec2.internal' and this object" podUID="45325b4d-ab21-46a5-a55d-336df593395b" pod="opendatahub/test-curl-7880-341e30"
Sep 16 12:26:50.633437 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.633404    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"]
Sep 16 12:26:50.633933 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.633914    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fd5535a-2560-44ff-8ff0-861a68a48a02" containerName="test-curl-7880-9b36a1"
Sep 16 12:26:50.633933 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.633934    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fd5535a-2560-44ff-8ff0-861a68a48a02" containerName="test-curl-7880-9b36a1"
Sep 16 12:26:50.634090 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.634058    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fd5535a-2560-44ff-8ff0-861a68a48a02" containerName="test-curl-7880-9b36a1"
Sep 16 12:26:50.636790 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.636769    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.639115 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.639093    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d5891124-openshift-default-dockercfg-x8b5b\""
Sep 16 12:26:50.646414 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.646389    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"]
Sep 16 12:26:50.662238 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:26:50.662203    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7880-341e30\" is forbidden: User \"system:node:ip-10-0-137-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-30.ec2.internal' and this object" podUID="45325b4d-ab21-46a5-a55d-336df593395b" pod="opendatahub/test-curl-7880-341e30"
Sep 16 12:26:50.735527 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.735475    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30e0588a-f0b0-425c-9c5c-cb41c5696981-istiod-ca-cert\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.735738 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.735542    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30e0588a-f0b0-425c-9c5c-cb41c5696981-istio-token\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.735738 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.735583    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-workload-certs\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.735738 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.735620    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-istio-envoy\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.735738 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.735683    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btdb2\" (UniqueName: \"kubernetes.io/projected/30e0588a-f0b0-425c-9c5c-cb41c5696981-kube-api-access-btdb2\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.735738 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.735720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-istio-data\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.736049 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.735819    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30e0588a-f0b0-425c-9c5c-cb41c5696981-istio-podinfo\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.736049 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.735899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-workload-socket\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.736049 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.735930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-credential-socket\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.781281 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.781248    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45325b4d-ab21-46a5-a55d-336df593395b" path="/var/lib/kubelet/pods/45325b4d-ab21-46a5-a55d-336df593395b/volumes"
Sep 16 12:26:50.837404 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.837348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-workload-certs\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.837404 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.837411    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-istio-envoy\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.837712 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.837433    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btdb2\" (UniqueName: \"kubernetes.io/projected/30e0588a-f0b0-425c-9c5c-cb41c5696981-kube-api-access-btdb2\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.837712 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.837456    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-istio-data\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.837712 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.837516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30e0588a-f0b0-425c-9c5c-cb41c5696981-istio-podinfo\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.837712 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.837583    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-workload-socket\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.837712 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.837612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-credential-socket\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.837712 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.837698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30e0588a-f0b0-425c-9c5c-cb41c5696981-istiod-ca-cert\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.838099 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.837737    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30e0588a-f0b0-425c-9c5c-cb41c5696981-istio-token\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.838099 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.837855    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-workload-certs\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.838099 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.837942    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-istio-data\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.838225 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.838088    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-workload-socket\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.838270 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.838217    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-credential-socket\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.838435 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.838419    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30e0588a-f0b0-425c-9c5c-cb41c5696981-istiod-ca-cert\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.839766 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.839743    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-istio-envoy\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.840020 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.840001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30e0588a-f0b0-425c-9c5c-cb41c5696981-istio-podinfo\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.844842 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.844820    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30e0588a-f0b0-425c-9c5c-cb41c5696981-istio-token\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.844995 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.844977    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btdb2\" (UniqueName: \"kubernetes.io/projected/30e0588a-f0b0-425c-9c5c-cb41c5696981-kube-api-access-btdb2\") pod \"e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") " pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:50.949209 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:50.949117    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:51.226496 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:51.226467    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"]
Sep 16 12:26:51.227381 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:26:51.227358    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e0588a_f0b0_425c_9c5c_cb41c5696981.slice/crio-3fdb89dbe2742426bc316f0e5a50389f742b54a254e3e8e72a63d255a575c19d WatchSource:0}: Error finding container 3fdb89dbe2742426bc316f0e5a50389f742b54a254e3e8e72a63d255a575c19d: Status 404 returned error can't find the container with id 3fdb89dbe2742426bc316f0e5a50389f742b54a254e3e8e72a63d255a575c19d
Sep 16 12:26:51.230138 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:51.230104    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:26:51.230238 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:51.230172    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:26:51.230238 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:51.230200    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:26:51.258957 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:51.258926    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9" event={"ID":"30e0588a-f0b0-425c-9c5c-cb41c5696981","Type":"ContainerStarted","Data":"3fdb89dbe2742426bc316f0e5a50389f742b54a254e3e8e72a63d255a575c19d"}
Sep 16 12:26:51.259869 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:51.259847    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7880-9b36a1" podUID="5fd5535a-2560-44ff-8ff0-861a68a48a02" containerName="test-curl-7880-9b36a1" containerID="cri-o://8b19420867bc411ac8d3f69ce223fe23cfe20f680a4336dd77da00adcbde2f13" gracePeriod=2
Sep 16 12:26:51.422063 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:51.422035    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7880-1e55d1"
Sep 16 12:26:51.546159 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:51.546062    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbjnf\" (UniqueName: \"kubernetes.io/projected/08b1cbdc-9c65-4304-b845-c54891361d2a-kube-api-access-pbjnf\") pod \"08b1cbdc-9c65-4304-b845-c54891361d2a\" (UID: \"08b1cbdc-9c65-4304-b845-c54891361d2a\") "
Sep 16 12:26:51.548154 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:51.548122    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08b1cbdc-9c65-4304-b845-c54891361d2a-kube-api-access-pbjnf" (OuterVolumeSpecName: "kube-api-access-pbjnf") pod "08b1cbdc-9c65-4304-b845-c54891361d2a" (UID: "08b1cbdc-9c65-4304-b845-c54891361d2a"). InnerVolumeSpecName "kube-api-access-pbjnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:26:51.647330 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:51.647294    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbjnf\" (UniqueName: \"kubernetes.io/projected/08b1cbdc-9c65-4304-b845-c54891361d2a-kube-api-access-pbjnf\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:26:52.267347 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:52.267250    2579 generic.go:356] "Generic (PLEG): container finished" podID="08b1cbdc-9c65-4304-b845-c54891361d2a" containerID="4ad67f3896fdfadf31189c85b73ac44be9284447205d9bce2a87a6e4e3cd9003" exitCode=137
Sep 16 12:26:52.267347 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:52.267293    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7880-1e55d1"
Sep 16 12:26:52.267816 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:52.267349    2579 scope.go:117] "RemoveContainer" containerID="4ad67f3896fdfadf31189c85b73ac44be9284447205d9bce2a87a6e4e3cd9003"
Sep 16 12:26:52.269331 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:52.269303    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9" event={"ID":"30e0588a-f0b0-425c-9c5c-cb41c5696981","Type":"ContainerStarted","Data":"04d6e38555c514b90677995c2c4729d07000644caeb3a6cedda234466e7d7335"}
Sep 16 12:26:52.278298 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:52.278280    2579 scope.go:117] "RemoveContainer" containerID="4ad67f3896fdfadf31189c85b73ac44be9284447205d9bce2a87a6e4e3cd9003"
Sep 16 12:26:52.278571 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:26:52.278546    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad67f3896fdfadf31189c85b73ac44be9284447205d9bce2a87a6e4e3cd9003\": container with ID starting with 4ad67f3896fdfadf31189c85b73ac44be9284447205d9bce2a87a6e4e3cd9003 not found: ID does not exist" containerID="4ad67f3896fdfadf31189c85b73ac44be9284447205d9bce2a87a6e4e3cd9003"
Sep 16 12:26:52.278640 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:52.278582    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad67f3896fdfadf31189c85b73ac44be9284447205d9bce2a87a6e4e3cd9003"} err="failed to get container status \"4ad67f3896fdfadf31189c85b73ac44be9284447205d9bce2a87a6e4e3cd9003\": rpc error: code = NotFound desc = could not find container \"4ad67f3896fdfadf31189c85b73ac44be9284447205d9bce2a87a6e4e3cd9003\": container with ID starting with 4ad67f3896fdfadf31189c85b73ac44be9284447205d9bce2a87a6e4e3cd9003 not found: ID does not exist"
Sep 16 12:26:52.289506 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:26:52.289480    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7880-1e55d1\" is forbidden: User \"system:node:ip-10-0-137-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-30.ec2.internal' and this object" podUID="08b1cbdc-9c65-4304-b845-c54891361d2a" pod="opendatahub/test-curl-7880-1e55d1"
Sep 16 12:26:52.290067 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:52.290031    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9" podStartSLOduration=2.290018331 podStartE2EDuration="2.290018331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:26:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:26:52.287521809 +0000 UTC m=+1405.988593148" watchObservedRunningTime="2026-09-16 12:26:52.290018331 +0000 UTC m=+1405.991089656"
Sep 16 12:26:52.780885 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:52.780848    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08b1cbdc-9c65-4304-b845-c54891361d2a" path="/var/lib/kubelet/pods/08b1cbdc-9c65-4304-b845-c54891361d2a/volumes"
Sep 16 12:26:52.949359 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:52.949318    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:52.954577 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:52.954551    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:53.276526 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:53.276503    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:53.277638 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:53.277620    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:26:53.440737 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:53.440710    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7880-9b36a1"
Sep 16 12:26:53.567412 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:53.567314    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhjbd\" (UniqueName: \"kubernetes.io/projected/5fd5535a-2560-44ff-8ff0-861a68a48a02-kube-api-access-bhjbd\") pod \"5fd5535a-2560-44ff-8ff0-861a68a48a02\" (UID: \"5fd5535a-2560-44ff-8ff0-861a68a48a02\") "
Sep 16 12:26:53.569375 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:53.569348    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd5535a-2560-44ff-8ff0-861a68a48a02-kube-api-access-bhjbd" (OuterVolumeSpecName: "kube-api-access-bhjbd") pod "5fd5535a-2560-44ff-8ff0-861a68a48a02" (UID: "5fd5535a-2560-44ff-8ff0-861a68a48a02"). InnerVolumeSpecName "kube-api-access-bhjbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:26:53.668644 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:53.668610    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhjbd\" (UniqueName: \"kubernetes.io/projected/5fd5535a-2560-44ff-8ff0-861a68a48a02-kube-api-access-bhjbd\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:26:54.283780 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:54.283748    2579 generic.go:356] "Generic (PLEG): container finished" podID="5fd5535a-2560-44ff-8ff0-861a68a48a02" containerID="8b19420867bc411ac8d3f69ce223fe23cfe20f680a4336dd77da00adcbde2f13" exitCode=137
Sep 16 12:26:54.284204 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:54.283793    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7880-9b36a1"
Sep 16 12:26:54.284204 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:54.283857    2579 scope.go:117] "RemoveContainer" containerID="8b19420867bc411ac8d3f69ce223fe23cfe20f680a4336dd77da00adcbde2f13"
Sep 16 12:26:54.293556 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:54.293536    2579 scope.go:117] "RemoveContainer" containerID="8b19420867bc411ac8d3f69ce223fe23cfe20f680a4336dd77da00adcbde2f13"
Sep 16 12:26:54.293830 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:26:54.293811    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b19420867bc411ac8d3f69ce223fe23cfe20f680a4336dd77da00adcbde2f13\": container with ID starting with 8b19420867bc411ac8d3f69ce223fe23cfe20f680a4336dd77da00adcbde2f13 not found: ID does not exist" containerID="8b19420867bc411ac8d3f69ce223fe23cfe20f680a4336dd77da00adcbde2f13"
Sep 16 12:26:54.293881 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:54.293839    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b19420867bc411ac8d3f69ce223fe23cfe20f680a4336dd77da00adcbde2f13"} err="failed to get container status \"8b19420867bc411ac8d3f69ce223fe23cfe20f680a4336dd77da00adcbde2f13\": rpc error: code = NotFound desc = could not find container \"8b19420867bc411ac8d3f69ce223fe23cfe20f680a4336dd77da00adcbde2f13\": container with ID starting with 8b19420867bc411ac8d3f69ce223fe23cfe20f680a4336dd77da00adcbde2f13 not found: ID does not exist"
Sep 16 12:26:54.294832 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:26:54.294813    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7880-9b36a1\" is forbidden: User \"system:node:ip-10-0-137-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-30.ec2.internal' and this object" podUID="5fd5535a-2560-44ff-8ff0-861a68a48a02" pod="opendatahub/test-curl-7880-9b36a1"
Sep 16 12:26:54.780722 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:54.780690    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fd5535a-2560-44ff-8ff0-861a68a48a02" path="/var/lib/kubelet/pods/5fd5535a-2560-44ff-8ff0-861a68a48a02/volumes"
Sep 16 12:26:58.965271 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:58.965233    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"]
Sep 16 12:26:58.968051 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:58.967996    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"
Sep 16 12:26:58.970396 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:58.970374    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d5891124-serving-cert\""
Sep 16 12:26:58.984244 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:58.984217    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"]
Sep 16 12:26:59.020039 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:59.020000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-openshift-service-ca\") pod \"maas-api-e2e-mt-d5891124-7d95c945bd-wgd28\" (UID: \"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"
Sep 16 12:26:59.020219 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:59.020149    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xkcn\" (UniqueName: \"kubernetes.io/projected/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-kube-api-access-7xkcn\") pod \"maas-api-e2e-mt-d5891124-7d95c945bd-wgd28\" (UID: \"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"
Sep 16 12:26:59.020337 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:59.020318    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-metrics-certs\") pod \"maas-api-e2e-mt-d5891124-7d95c945bd-wgd28\" (UID: \"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"
Sep 16 12:26:59.020385 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:59.020374    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-maas-api-tls\") pod \"maas-api-e2e-mt-d5891124-7d95c945bd-wgd28\" (UID: \"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"
Sep 16 12:26:59.121944 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:59.121907    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xkcn\" (UniqueName: \"kubernetes.io/projected/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-kube-api-access-7xkcn\") pod \"maas-api-e2e-mt-d5891124-7d95c945bd-wgd28\" (UID: \"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"
Sep 16 12:26:59.122148 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:59.121970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-metrics-certs\") pod \"maas-api-e2e-mt-d5891124-7d95c945bd-wgd28\" (UID: \"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"
Sep 16 12:26:59.122148 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:59.122031    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-maas-api-tls\") pod \"maas-api-e2e-mt-d5891124-7d95c945bd-wgd28\" (UID: \"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"
Sep 16 12:26:59.122148 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:59.122094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-openshift-service-ca\") pod \"maas-api-e2e-mt-d5891124-7d95c945bd-wgd28\" (UID: \"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"
Sep 16 12:26:59.122331 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:26:59.122151    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-serving-cert: secret "maas-api-e2e-mt-d5891124-serving-cert" not found
Sep 16 12:26:59.122331 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:26:59.122227    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-maas-api-tls podName:c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12 nodeName:}" failed. No retries permitted until 2026-09-16 12:26:59.622204667 +0000 UTC m=+1413.323275972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-maas-api-tls") pod "maas-api-e2e-mt-d5891124-7d95c945bd-wgd28" (UID: "c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12") : secret "maas-api-e2e-mt-d5891124-serving-cert" not found
Sep 16 12:26:59.122755 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:59.122733    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-openshift-service-ca\") pod \"maas-api-e2e-mt-d5891124-7d95c945bd-wgd28\" (UID: \"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"
Sep 16 12:26:59.124454 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:59.124433    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-metrics-certs\") pod \"maas-api-e2e-mt-d5891124-7d95c945bd-wgd28\" (UID: \"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"
Sep 16 12:26:59.135490 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:59.135467    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xkcn\" (UniqueName: \"kubernetes.io/projected/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-kube-api-access-7xkcn\") pod \"maas-api-e2e-mt-d5891124-7d95c945bd-wgd28\" (UID: \"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"
Sep 16 12:26:59.627619 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:26:59.627583    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-maas-api-tls\") pod \"maas-api-e2e-mt-d5891124-7d95c945bd-wgd28\" (UID: \"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"
Sep 16 12:26:59.627840 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:26:59.627812    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-serving-cert: secret "maas-api-e2e-mt-d5891124-serving-cert" not found
Sep 16 12:26:59.627944 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:26:59.627918    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-maas-api-tls podName:c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12 nodeName:}" failed. No retries permitted until 2026-09-16 12:27:00.627896774 +0000 UTC m=+1414.328968099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-maas-api-tls") pod "maas-api-e2e-mt-d5891124-7d95c945bd-wgd28" (UID: "c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12") : secret "maas-api-e2e-mt-d5891124-serving-cert" not found
Sep 16 12:27:00.637929 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:00.637894    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-maas-api-tls\") pod \"maas-api-e2e-mt-d5891124-7d95c945bd-wgd28\" (UID: \"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"
Sep 16 12:27:00.640417 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:00.640384    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-maas-api-tls\") pod \"maas-api-e2e-mt-d5891124-7d95c945bd-wgd28\" (UID: \"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"
Sep 16 12:27:00.780986 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:00.780953    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"
Sep 16 12:27:01.057329 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:01.057303    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"]
Sep 16 12:27:01.058580 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:27:01.058551    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c9272c_a6fb_47f4_bed8_9e13c2e1ae12.slice/crio-51caafc08ebeddd928aaa865848f37fa5ecf46c04e7a98ac30baac99435bf0c3 WatchSource:0}: Error finding container 51caafc08ebeddd928aaa865848f37fa5ecf46c04e7a98ac30baac99435bf0c3: Status 404 returned error can't find the container with id 51caafc08ebeddd928aaa865848f37fa5ecf46c04e7a98ac30baac99435bf0c3
Sep 16 12:27:01.334859 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:01.334822    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28" event={"ID":"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12","Type":"ContainerStarted","Data":"51caafc08ebeddd928aaa865848f37fa5ecf46c04e7a98ac30baac99435bf0c3"}
Sep 16 12:27:02.343111 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:02.343071    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28" event={"ID":"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12","Type":"ContainerStarted","Data":"000476e771b3005d991e5f7a17f54c0d1831250bb48afbf00562ec356e9a802a"}
Sep 16 12:27:02.343493 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:02.343155    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"
Sep 16 12:27:02.363753 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:02.363699    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28" podStartSLOduration=3.844279251 podStartE2EDuration="4.363683826s" totalImagesPullingTime="519.404575ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:27:02.361093595 +0000 UTC m=+1416.062164998" watchObservedRunningTime="2026-09-16 12:27:02.363683826 +0000 UTC m=+1416.064755152"
Sep 16 12:27:08.354906 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:08.354877    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"
Sep 16 12:27:15.880621 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:15.880583    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:27:24.317866 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:24.317816    2579 patch_prober.go:28] interesting pod/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 12:27:24.318264 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:24.317916    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9" podUID="30e0588a-f0b0-425c-9c5c-cb41c5696981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 12:27:25.762593 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:25.760295    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"]
Sep 16 12:27:25.762593 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:25.760550    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx" podUID="f35bc2fb-f32f-4f0f-8e78-918f927616e3" containerName="istio-proxy" containerID="cri-o://9b701cefee6e2c48aec2a9a2f3d11ae125f4d01ffe043e25fdafa393df2fb446" gracePeriod=30
Sep 16 12:27:26.500871 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:26.500835    2579 generic.go:356] "Generic (PLEG): container finished" podID="30e0588a-f0b0-425c-9c5c-cb41c5696981" containerID="04d6e38555c514b90677995c2c4729d07000644caeb3a6cedda234466e7d7335" exitCode=0
Sep 16 12:27:26.501058 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:26.500918    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9" event={"ID":"30e0588a-f0b0-425c-9c5c-cb41c5696981","Type":"ContainerDied","Data":"04d6e38555c514b90677995c2c4729d07000644caeb3a6cedda234466e7d7335"}
Sep 16 12:27:26.501363 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:26.501348    2579 scope.go:117] "RemoveContainer" containerID="04d6e38555c514b90677995c2c4729d07000644caeb3a6cedda234466e7d7335"
Sep 16 12:27:26.502601 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:26.502570    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:27:26.502746 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:26.502673    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:27:26.502746 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:26.502722    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:27:26.922070 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:26.922039    2579 scope.go:117] "RemoveContainer" containerID="19afd3b3595cd4df0f9978f897ca3077edb04ef0eb54b065994a8f53ac5744f6"
Sep 16 12:27:27.510181 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:27.510141    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9" event={"ID":"30e0588a-f0b0-425c-9c5c-cb41c5696981","Type":"ContainerStarted","Data":"5de4efdf7fcb442d532747986e8f2b27514a5a645a2ddd2edfcee45625d0f65c"}
Sep 16 12:27:27.949726 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:27.949691    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:27:27.950119 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:27.949917    2579 patch_prober.go:28] interesting pod/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 16 12:27:27.950119 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:27.949959    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9" podUID="30e0588a-f0b0-425c-9c5c-cb41c5696981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 16 12:27:28.950057 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:28.950016    2579 patch_prober.go:28] interesting pod/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 16 12:27:28.950524 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:28.950076    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9" podUID="30e0588a-f0b0-425c-9c5c-cb41c5696981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 16 12:27:29.950100 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:29.950068    2579 patch_prober.go:28] interesting pod/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 16 12:27:29.950475 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:29.950117    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9" podUID="30e0588a-f0b0-425c-9c5c-cb41c5696981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 16 12:27:30.950511 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:30.950130    2579 patch_prober.go:28] interesting pod/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 16 12:27:30.950511 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:30.950186    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9" podUID="30e0588a-f0b0-425c-9c5c-cb41c5696981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 16 12:27:31.063295 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.063261    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:27:31.155299 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.155262    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istio-podinfo\") pod \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") "
Sep 16 12:27:31.155480 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.155336    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlqwv\" (UniqueName: \"kubernetes.io/projected/f35bc2fb-f32f-4f0f-8e78-918f927616e3-kube-api-access-rlqwv\") pod \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") "
Sep 16 12:27:31.155480 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.155362    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istio-token\") pod \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") "
Sep 16 12:27:31.155480 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.155385    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-credential-socket\") pod \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") "
Sep 16 12:27:31.155480 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.155439    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istio-data\") pod \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") "
Sep 16 12:27:31.155480 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.155453    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istio-envoy\") pod \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") "
Sep 16 12:27:31.155480 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.155469    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istiod-ca-cert\") pod \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") "
Sep 16 12:27:31.155700 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.155506    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-workload-certs\") pod \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") "
Sep 16 12:27:31.155700 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.155537    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-workload-socket\") pod \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\" (UID: \"f35bc2fb-f32f-4f0f-8e78-918f927616e3\") "
Sep 16 12:27:31.156370 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.156047    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f35bc2fb-f32f-4f0f-8e78-918f927616e3" (UID: "f35bc2fb-f32f-4f0f-8e78-918f927616e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:31.156370 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.156103    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f35bc2fb-f32f-4f0f-8e78-918f927616e3" (UID: "f35bc2fb-f32f-4f0f-8e78-918f927616e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:31.156370 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.156292    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f35bc2fb-f32f-4f0f-8e78-918f927616e3" (UID: "f35bc2fb-f32f-4f0f-8e78-918f927616e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:27:31.156370 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.156318    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istio-data" (OuterVolumeSpecName: "istio-data") pod "f35bc2fb-f32f-4f0f-8e78-918f927616e3" (UID: "f35bc2fb-f32f-4f0f-8e78-918f927616e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:31.156694 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.156466    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f35bc2fb-f32f-4f0f-8e78-918f927616e3" (UID: "f35bc2fb-f32f-4f0f-8e78-918f927616e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:31.157989 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.157962    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f35bc2fb-f32f-4f0f-8e78-918f927616e3" (UID: "f35bc2fb-f32f-4f0f-8e78-918f927616e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:31.158273 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.158244    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f35bc2fb-f32f-4f0f-8e78-918f927616e3" (UID: "f35bc2fb-f32f-4f0f-8e78-918f927616e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:27:31.158453 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.158426    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istio-token" (OuterVolumeSpecName: "istio-token") pod "f35bc2fb-f32f-4f0f-8e78-918f927616e3" (UID: "f35bc2fb-f32f-4f0f-8e78-918f927616e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:27:31.158453 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.158438    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f35bc2fb-f32f-4f0f-8e78-918f927616e3-kube-api-access-rlqwv" (OuterVolumeSpecName: "kube-api-access-rlqwv") pod "f35bc2fb-f32f-4f0f-8e78-918f927616e3" (UID: "f35bc2fb-f32f-4f0f-8e78-918f927616e3"). InnerVolumeSpecName "kube-api-access-rlqwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:27:31.257284 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.257247    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlqwv\" (UniqueName: \"kubernetes.io/projected/f35bc2fb-f32f-4f0f-8e78-918f927616e3-kube-api-access-rlqwv\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:27:31.257284 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.257282    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istio-token\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:27:31.257284 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.257291    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-credential-socket\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:27:31.257524 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.257299    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istio-data\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:27:31.257524 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.257309    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istio-envoy\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:27:31.257524 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.257317    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istiod-ca-cert\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:27:31.257524 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.257325    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-workload-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:27:31.257524 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.257332    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f35bc2fb-f32f-4f0f-8e78-918f927616e3-workload-socket\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:27:31.257524 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.257339    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f35bc2fb-f32f-4f0f-8e78-918f927616e3-istio-podinfo\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:27:31.536886 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.536847    2579 generic.go:356] "Generic (PLEG): container finished" podID="f35bc2fb-f32f-4f0f-8e78-918f927616e3" containerID="9b701cefee6e2c48aec2a9a2f3d11ae125f4d01ffe043e25fdafa393df2fb446" exitCode=0
Sep 16 12:27:31.537071 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.536902    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"
Sep 16 12:27:31.537071 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.536937    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx" event={"ID":"f35bc2fb-f32f-4f0f-8e78-918f927616e3","Type":"ContainerDied","Data":"9b701cefee6e2c48aec2a9a2f3d11ae125f4d01ffe043e25fdafa393df2fb446"}
Sep 16 12:27:31.537071 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.537005    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx" event={"ID":"f35bc2fb-f32f-4f0f-8e78-918f927616e3","Type":"ContainerDied","Data":"0005541f307559c3ca7149bb87bd1cd25d5c63761be9f4d94ac4992206031c5e"}
Sep 16 12:27:31.537071 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.537027    2579 scope.go:117] "RemoveContainer" containerID="9b701cefee6e2c48aec2a9a2f3d11ae125f4d01ffe043e25fdafa393df2fb446"
Sep 16 12:27:31.548588 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.548568    2579 scope.go:117] "RemoveContainer" containerID="255281c2fb0468af70a0473be5d68852a8712a74f640af6ef45ea760c676c037"
Sep 16 12:27:31.557632 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.557612    2579 scope.go:117] "RemoveContainer" containerID="9b701cefee6e2c48aec2a9a2f3d11ae125f4d01ffe043e25fdafa393df2fb446"
Sep 16 12:27:31.558030 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:27:31.558002    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b701cefee6e2c48aec2a9a2f3d11ae125f4d01ffe043e25fdafa393df2fb446\": container with ID starting with 9b701cefee6e2c48aec2a9a2f3d11ae125f4d01ffe043e25fdafa393df2fb446 not found: ID does not exist" containerID="9b701cefee6e2c48aec2a9a2f3d11ae125f4d01ffe043e25fdafa393df2fb446"
Sep 16 12:27:31.558112 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.558044    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b701cefee6e2c48aec2a9a2f3d11ae125f4d01ffe043e25fdafa393df2fb446"} err="failed to get container status \"9b701cefee6e2c48aec2a9a2f3d11ae125f4d01ffe043e25fdafa393df2fb446\": rpc error: code = NotFound desc = could not find container \"9b701cefee6e2c48aec2a9a2f3d11ae125f4d01ffe043e25fdafa393df2fb446\": container with ID starting with 9b701cefee6e2c48aec2a9a2f3d11ae125f4d01ffe043e25fdafa393df2fb446 not found: ID does not exist"
Sep 16 12:27:31.558112 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.558070    2579 scope.go:117] "RemoveContainer" containerID="255281c2fb0468af70a0473be5d68852a8712a74f640af6ef45ea760c676c037"
Sep 16 12:27:31.558394 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:27:31.558371    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"255281c2fb0468af70a0473be5d68852a8712a74f640af6ef45ea760c676c037\": container with ID starting with 255281c2fb0468af70a0473be5d68852a8712a74f640af6ef45ea760c676c037 not found: ID does not exist" containerID="255281c2fb0468af70a0473be5d68852a8712a74f640af6ef45ea760c676c037"
Sep 16 12:27:31.558452 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.558403    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"255281c2fb0468af70a0473be5d68852a8712a74f640af6ef45ea760c676c037"} err="failed to get container status \"255281c2fb0468af70a0473be5d68852a8712a74f640af6ef45ea760c676c037\": rpc error: code = NotFound desc = could not find container \"255281c2fb0468af70a0473be5d68852a8712a74f640af6ef45ea760c676c037\": container with ID starting with 255281c2fb0468af70a0473be5d68852a8712a74f640af6ef45ea760c676c037 not found: ID does not exist"
Sep 16 12:27:31.561728 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.561619    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"]
Sep 16 12:27:31.564589 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.564570    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f79e22-openshift-default-6b95ddbbfb-6xxlx"]
Sep 16 12:27:31.950092 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.950041    2579 patch_prober.go:28] interesting pod/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 16 12:27:31.950264 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:31.950110    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9" podUID="30e0588a-f0b0-425c-9c5c-cb41c5696981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 16 12:27:32.781564 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:32.781528    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f35bc2fb-f32f-4f0f-8e78-918f927616e3" path="/var/lib/kubelet/pods/f35bc2fb-f32f-4f0f-8e78-918f927616e3/volumes"
Sep 16 12:27:32.953528 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:32.953499    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:27:32.953862 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:32.953838    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:27:32.954518 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:32.954495    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:27:40.029108 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:40.029070    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"]
Sep 16 12:27:44.567673 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:44.567634    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:27:55.049118 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.049079    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp" podUID="a40af25e-97c6-4425-8256-e07f985836b5" containerName="main" containerID="cri-o://66069bedc803016db7cfcecceeea25a6626526dd113c8743949e4370f4aabb51" gracePeriod=45
Sep 16 12:27:55.308034 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.307974    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:27:55.398041 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.398005    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmmvt\" (UniqueName: \"kubernetes.io/projected/a40af25e-97c6-4425-8256-e07f985836b5-kube-api-access-cmmvt\") pod \"a40af25e-97c6-4425-8256-e07f985836b5\" (UID: \"a40af25e-97c6-4425-8256-e07f985836b5\") "
Sep 16 12:27:55.398232 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.398047    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a40af25e-97c6-4425-8256-e07f985836b5-tmp-dir\") pod \"a40af25e-97c6-4425-8256-e07f985836b5\" (UID: \"a40af25e-97c6-4425-8256-e07f985836b5\") "
Sep 16 12:27:55.398232 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.398086    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a40af25e-97c6-4425-8256-e07f985836b5-dshm\") pod \"a40af25e-97c6-4425-8256-e07f985836b5\" (UID: \"a40af25e-97c6-4425-8256-e07f985836b5\") "
Sep 16 12:27:55.398232 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.398114    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a40af25e-97c6-4425-8256-e07f985836b5-tls-certs\") pod \"a40af25e-97c6-4425-8256-e07f985836b5\" (UID: \"a40af25e-97c6-4425-8256-e07f985836b5\") "
Sep 16 12:27:55.398232 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.398139    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a40af25e-97c6-4425-8256-e07f985836b5-home\") pod \"a40af25e-97c6-4425-8256-e07f985836b5\" (UID: \"a40af25e-97c6-4425-8256-e07f985836b5\") "
Sep 16 12:27:55.398443 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.398330    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a40af25e-97c6-4425-8256-e07f985836b5-model-cache\") pod \"a40af25e-97c6-4425-8256-e07f985836b5\" (UID: \"a40af25e-97c6-4425-8256-e07f985836b5\") "
Sep 16 12:27:55.398443 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.398407    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40af25e-97c6-4425-8256-e07f985836b5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a40af25e-97c6-4425-8256-e07f985836b5" (UID: "a40af25e-97c6-4425-8256-e07f985836b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:55.398558 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.398458    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40af25e-97c6-4425-8256-e07f985836b5-home" (OuterVolumeSpecName: "home") pod "a40af25e-97c6-4425-8256-e07f985836b5" (UID: "a40af25e-97c6-4425-8256-e07f985836b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:55.398718 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.398694    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40af25e-97c6-4425-8256-e07f985836b5-model-cache" (OuterVolumeSpecName: "model-cache") pod "a40af25e-97c6-4425-8256-e07f985836b5" (UID: "a40af25e-97c6-4425-8256-e07f985836b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:55.398968 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.398948    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a40af25e-97c6-4425-8256-e07f985836b5-model-cache\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:27:55.399110 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.398974    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a40af25e-97c6-4425-8256-e07f985836b5-tmp-dir\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:27:55.399110 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.398989    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a40af25e-97c6-4425-8256-e07f985836b5-home\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:27:55.400436 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.400410    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a40af25e-97c6-4425-8256-e07f985836b5-dshm" (OuterVolumeSpecName: "dshm") pod "a40af25e-97c6-4425-8256-e07f985836b5" (UID: "a40af25e-97c6-4425-8256-e07f985836b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:27:55.400529 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.400468    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a40af25e-97c6-4425-8256-e07f985836b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a40af25e-97c6-4425-8256-e07f985836b5" (UID: "a40af25e-97c6-4425-8256-e07f985836b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:27:55.400529 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.400505    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a40af25e-97c6-4425-8256-e07f985836b5-kube-api-access-cmmvt" (OuterVolumeSpecName: "kube-api-access-cmmvt") pod "a40af25e-97c6-4425-8256-e07f985836b5" (UID: "a40af25e-97c6-4425-8256-e07f985836b5"). InnerVolumeSpecName "kube-api-access-cmmvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:27:55.499506 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.499473    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmmvt\" (UniqueName: \"kubernetes.io/projected/a40af25e-97c6-4425-8256-e07f985836b5-kube-api-access-cmmvt\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:27:55.499506 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.499498    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a40af25e-97c6-4425-8256-e07f985836b5-dshm\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:27:55.499726 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.499520    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a40af25e-97c6-4425-8256-e07f985836b5-tls-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:27:55.690254 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.690213    2579 generic.go:356] "Generic (PLEG): container finished" podID="a40af25e-97c6-4425-8256-e07f985836b5" containerID="66069bedc803016db7cfcecceeea25a6626526dd113c8743949e4370f4aabb51" exitCode=0
Sep 16 12:27:55.690466 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.690257    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"
Sep 16 12:27:55.690466 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.690300    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp" event={"ID":"a40af25e-97c6-4425-8256-e07f985836b5","Type":"ContainerDied","Data":"66069bedc803016db7cfcecceeea25a6626526dd113c8743949e4370f4aabb51"}
Sep 16 12:27:55.690466 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.690349    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp" event={"ID":"a40af25e-97c6-4425-8256-e07f985836b5","Type":"ContainerDied","Data":"178026a625e71fc89d6860b04c2303cde6041196f92c40e331a95b7f071d6ce9"}
Sep 16 12:27:55.690466 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.690369    2579 scope.go:117] "RemoveContainer" containerID="66069bedc803016db7cfcecceeea25a6626526dd113c8743949e4370f4aabb51"
Sep 16 12:27:55.700826 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.700808    2579 scope.go:117] "RemoveContainer" containerID="66069bedc803016db7cfcecceeea25a6626526dd113c8743949e4370f4aabb51"
Sep 16 12:27:55.701087 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:27:55.701071    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66069bedc803016db7cfcecceeea25a6626526dd113c8743949e4370f4aabb51\": container with ID starting with 66069bedc803016db7cfcecceeea25a6626526dd113c8743949e4370f4aabb51 not found: ID does not exist" containerID="66069bedc803016db7cfcecceeea25a6626526dd113c8743949e4370f4aabb51"
Sep 16 12:27:55.701124 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.701094    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66069bedc803016db7cfcecceeea25a6626526dd113c8743949e4370f4aabb51"} err="failed to get container status \"66069bedc803016db7cfcecceeea25a6626526dd113c8743949e4370f4aabb51\": rpc error: code = NotFound desc = could not find container \"66069bedc803016db7cfcecceeea25a6626526dd113c8743949e4370f4aabb51\": container with ID starting with 66069bedc803016db7cfcecceeea25a6626526dd113c8743949e4370f4aabb51 not found: ID does not exist"
Sep 16 12:27:55.725246 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.725213    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"]
Sep 16 12:27:55.731754 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:55.731719    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ef3317/test-model-ef3317-kserve-7866bc7f79-bbtnp"]
Sep 16 12:27:56.783264 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:27:56.783224    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a40af25e-97c6-4425-8256-e07f985836b5" path="/var/lib/kubelet/pods/a40af25e-97c6-4425-8256-e07f985836b5/volumes"
Sep 16 12:28:10.960100 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:10.960056    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:28:19.450046 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:19.450012    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"]
Sep 16 12:28:19.450606 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:19.450255    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7" podUID="2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1" containerName="istio-proxy" containerID="cri-o://d4025312c43c98c1c39d1dd55e577a8879ad252f026aa1a5c0d066d9c307d25e" gracePeriod=30
Sep 16 12:28:24.709168 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.709132    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:28:24.814039 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.813965    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istiod-ca-cert\") pod \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") "
Sep 16 12:28:24.814175 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.814062    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-workload-certs\") pod \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") "
Sep 16 12:28:24.814175 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.814085    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istio-envoy\") pod \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") "
Sep 16 12:28:24.814175 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.814135    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-credential-socket\") pod \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") "
Sep 16 12:28:24.814328 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.814185    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-workload-socket\") pod \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") "
Sep 16 12:28:24.814328 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.814229    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tc5kp\" (UniqueName: \"kubernetes.io/projected/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-kube-api-access-tc5kp\") pod \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") "
Sep 16 12:28:24.814328 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.814288    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istio-data\") pod \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") "
Sep 16 12:28:24.814488 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.814325    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istio-token\") pod \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") "
Sep 16 12:28:24.814488 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.814354    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istio-podinfo\") pod \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\" (UID: \"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1\") "
Sep 16 12:28:24.814488 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.814392    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1" (UID: "2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:24.814488 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.814430    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1" (UID: "2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:28:24.814727 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.814550    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1" (UID: "2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:24.814727 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.814566    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1" (UID: "2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:24.814727 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.814712    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istio-data" (OuterVolumeSpecName: "istio-data") pod "2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1" (UID: "2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:24.814879 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.814804    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-workload-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:28:24.814879 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.814826    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-credential-socket\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:28:24.814879 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.814841    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-workload-socket\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:28:24.814879 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.814854    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istio-data\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:28:24.814879 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.814870    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istiod-ca-cert\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:28:24.816336 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.816313    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1" (UID: "2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:28:24.816647 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.816627    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-kube-api-access-tc5kp" (OuterVolumeSpecName: "kube-api-access-tc5kp") pod "2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1" (UID: "2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1"). InnerVolumeSpecName "kube-api-access-tc5kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:28:24.816863 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.816846    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1" (UID: "2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:28:24.816969 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.816952    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istio-token" (OuterVolumeSpecName: "istio-token") pod "2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1" (UID: "2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:28:24.865616 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.865589    2579 generic.go:356] "Generic (PLEG): container finished" podID="2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1" containerID="d4025312c43c98c1c39d1dd55e577a8879ad252f026aa1a5c0d066d9c307d25e" exitCode=0
Sep 16 12:28:24.865742 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.865639    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"
Sep 16 12:28:24.865742 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.865650    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7" event={"ID":"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1","Type":"ContainerDied","Data":"d4025312c43c98c1c39d1dd55e577a8879ad252f026aa1a5c0d066d9c307d25e"}
Sep 16 12:28:24.865742 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.865707    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7" event={"ID":"2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1","Type":"ContainerDied","Data":"d48b43954a1e4025e980d6e37b391724896169f1b0f2b70caaab8c299f6623a0"}
Sep 16 12:28:24.865742 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.865729    2579 scope.go:117] "RemoveContainer" containerID="d4025312c43c98c1c39d1dd55e577a8879ad252f026aa1a5c0d066d9c307d25e"
Sep 16 12:28:24.875880 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.875863    2579 scope.go:117] "RemoveContainer" containerID="9b14644d0cf41af7fcc4a9a4d4ea4db3e6641769c585586246def8ff6e393d00"
Sep 16 12:28:24.884358 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.884341    2579 scope.go:117] "RemoveContainer" containerID="d4025312c43c98c1c39d1dd55e577a8879ad252f026aa1a5c0d066d9c307d25e"
Sep 16 12:28:24.884604 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:28:24.884585    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4025312c43c98c1c39d1dd55e577a8879ad252f026aa1a5c0d066d9c307d25e\": container with ID starting with d4025312c43c98c1c39d1dd55e577a8879ad252f026aa1a5c0d066d9c307d25e not found: ID does not exist" containerID="d4025312c43c98c1c39d1dd55e577a8879ad252f026aa1a5c0d066d9c307d25e"
Sep 16 12:28:24.884702 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.884611    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4025312c43c98c1c39d1dd55e577a8879ad252f026aa1a5c0d066d9c307d25e"} err="failed to get container status \"d4025312c43c98c1c39d1dd55e577a8879ad252f026aa1a5c0d066d9c307d25e\": rpc error: code = NotFound desc = could not find container \"d4025312c43c98c1c39d1dd55e577a8879ad252f026aa1a5c0d066d9c307d25e\": container with ID starting with d4025312c43c98c1c39d1dd55e577a8879ad252f026aa1a5c0d066d9c307d25e not found: ID does not exist"
Sep 16 12:28:24.884702 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.884627    2579 scope.go:117] "RemoveContainer" containerID="9b14644d0cf41af7fcc4a9a4d4ea4db3e6641769c585586246def8ff6e393d00"
Sep 16 12:28:24.884878 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:28:24.884863    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b14644d0cf41af7fcc4a9a4d4ea4db3e6641769c585586246def8ff6e393d00\": container with ID starting with 9b14644d0cf41af7fcc4a9a4d4ea4db3e6641769c585586246def8ff6e393d00 not found: ID does not exist" containerID="9b14644d0cf41af7fcc4a9a4d4ea4db3e6641769c585586246def8ff6e393d00"
Sep 16 12:28:24.884915 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.884882    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b14644d0cf41af7fcc4a9a4d4ea4db3e6641769c585586246def8ff6e393d00"} err="failed to get container status \"9b14644d0cf41af7fcc4a9a4d4ea4db3e6641769c585586246def8ff6e393d00\": rpc error: code = NotFound desc = could not find container \"9b14644d0cf41af7fcc4a9a4d4ea4db3e6641769c585586246def8ff6e393d00\": container with ID starting with 9b14644d0cf41af7fcc4a9a4d4ea4db3e6641769c585586246def8ff6e393d00 not found: ID does not exist"
Sep 16 12:28:24.889306 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.889281    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"]
Sep 16 12:28:24.891852 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.891830    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ef3317-openshift-default-5fc7758954-c5gn7"]
Sep 16 12:28:24.915589 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.915568    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istio-token\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:28:24.915589 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.915588    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istio-podinfo\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:28:24.915700 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.915597    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-istio-envoy\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:28:24.915700 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:24.915605    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc5kp\" (UniqueName: \"kubernetes.io/projected/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1-kube-api-access-tc5kp\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:28:26.781635 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:26.781602    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1" path="/var/lib/kubelet/pods/2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1/volumes"
Sep 16 12:28:26.969347 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:26.969313    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"]
Sep 16 12:28:26.969941 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:26.969920    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f35bc2fb-f32f-4f0f-8e78-918f927616e3" containerName="istio-proxy"
Sep 16 12:28:26.969941 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:26.969941    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f35bc2fb-f32f-4f0f-8e78-918f927616e3" containerName="istio-proxy"
Sep 16 12:28:26.970122 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:26.969966    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1" containerName="istio-proxy"
Sep 16 12:28:26.970122 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:26.969975    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1" containerName="istio-proxy"
Sep 16 12:28:26.970122 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:26.969989    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1" containerName="istio-proxy"
Sep 16 12:28:26.970122 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:26.969997    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1" containerName="istio-proxy"
Sep 16 12:28:26.970122 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:26.970022    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a40af25e-97c6-4425-8256-e07f985836b5" containerName="main"
Sep 16 12:28:26.970122 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:26.970030    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a40af25e-97c6-4425-8256-e07f985836b5" containerName="main"
Sep 16 12:28:26.970381 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:26.970146    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1" containerName="istio-proxy"
Sep 16 12:28:26.970381 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:26.970159    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a40af25e-97c6-4425-8256-e07f985836b5" containerName="main"
Sep 16 12:28:26.970381 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:26.970168    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f35bc2fb-f32f-4f0f-8e78-918f927616e3" containerName="istio-proxy"
Sep 16 12:28:26.973359 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:26.973328    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"
Sep 16 12:28:26.975995 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:26.975960    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-0a9008-serving-cert\""
Sep 16 12:28:26.982149 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:26.982121    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"]
Sep 16 12:28:27.033222 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:27.033135    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ba65312-d842-40fc-be8a-5ef3fb1eca79-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4\" (UID: \"1ba65312-d842-40fc-be8a-5ef3fb1eca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"
Sep 16 12:28:27.033222 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:27.033181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ba65312-d842-40fc-be8a-5ef3fb1eca79-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4\" (UID: \"1ba65312-d842-40fc-be8a-5ef3fb1eca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"
Sep 16 12:28:27.033395 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:27.033237    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ba65312-d842-40fc-be8a-5ef3fb1eca79-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4\" (UID: \"1ba65312-d842-40fc-be8a-5ef3fb1eca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"
Sep 16 12:28:27.033395 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:27.033270    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbztf\" (UniqueName: \"kubernetes.io/projected/1ba65312-d842-40fc-be8a-5ef3fb1eca79-kube-api-access-tbztf\") pod \"maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4\" (UID: \"1ba65312-d842-40fc-be8a-5ef3fb1eca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"
Sep 16 12:28:27.134185 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:27.134150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ba65312-d842-40fc-be8a-5ef3fb1eca79-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4\" (UID: \"1ba65312-d842-40fc-be8a-5ef3fb1eca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"
Sep 16 12:28:27.134185 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:27.134190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ba65312-d842-40fc-be8a-5ef3fb1eca79-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4\" (UID: \"1ba65312-d842-40fc-be8a-5ef3fb1eca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"
Sep 16 12:28:27.134438 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:27.134220    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ba65312-d842-40fc-be8a-5ef3fb1eca79-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4\" (UID: \"1ba65312-d842-40fc-be8a-5ef3fb1eca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"
Sep 16 12:28:27.134438 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:27.134267    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbztf\" (UniqueName: \"kubernetes.io/projected/1ba65312-d842-40fc-be8a-5ef3fb1eca79-kube-api-access-tbztf\") pod \"maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4\" (UID: \"1ba65312-d842-40fc-be8a-5ef3fb1eca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"
Sep 16 12:28:27.134438 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:28:27.134314    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-serving-cert: secret "maas-api-e2e-shared-a-w3-0a9008-serving-cert" not found
Sep 16 12:28:27.134438 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:28:27.134393    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ba65312-d842-40fc-be8a-5ef3fb1eca79-maas-api-tls podName:1ba65312-d842-40fc-be8a-5ef3fb1eca79 nodeName:}" failed. No retries permitted until 2026-09-16 12:28:27.634369167 +0000 UTC m=+1501.335440481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1ba65312-d842-40fc-be8a-5ef3fb1eca79-maas-api-tls") pod "maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4" (UID: "1ba65312-d842-40fc-be8a-5ef3fb1eca79") : secret "maas-api-e2e-shared-a-w3-0a9008-serving-cert" not found
Sep 16 12:28:27.134989 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:27.134966    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ba65312-d842-40fc-be8a-5ef3fb1eca79-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4\" (UID: \"1ba65312-d842-40fc-be8a-5ef3fb1eca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"
Sep 16 12:28:27.136705 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:27.136676    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ba65312-d842-40fc-be8a-5ef3fb1eca79-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4\" (UID: \"1ba65312-d842-40fc-be8a-5ef3fb1eca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"
Sep 16 12:28:27.142053 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:27.142031    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbztf\" (UniqueName: \"kubernetes.io/projected/1ba65312-d842-40fc-be8a-5ef3fb1eca79-kube-api-access-tbztf\") pod \"maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4\" (UID: \"1ba65312-d842-40fc-be8a-5ef3fb1eca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"
Sep 16 12:28:27.639432 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:27.639391    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ba65312-d842-40fc-be8a-5ef3fb1eca79-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4\" (UID: \"1ba65312-d842-40fc-be8a-5ef3fb1eca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"
Sep 16 12:28:27.639640 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:28:27.639592    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-serving-cert: secret "maas-api-e2e-shared-a-w3-0a9008-serving-cert" not found
Sep 16 12:28:27.639741 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:28:27.639699    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1ba65312-d842-40fc-be8a-5ef3fb1eca79-maas-api-tls podName:1ba65312-d842-40fc-be8a-5ef3fb1eca79 nodeName:}" failed. No retries permitted until 2026-09-16 12:28:28.639677034 +0000 UTC m=+1502.340748353 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1ba65312-d842-40fc-be8a-5ef3fb1eca79-maas-api-tls") pod "maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4" (UID: "1ba65312-d842-40fc-be8a-5ef3fb1eca79") : secret "maas-api-e2e-shared-a-w3-0a9008-serving-cert" not found
Sep 16 12:28:28.649100 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:28.649065    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ba65312-d842-40fc-be8a-5ef3fb1eca79-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4\" (UID: \"1ba65312-d842-40fc-be8a-5ef3fb1eca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"
Sep 16 12:28:28.651551 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:28.651525    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ba65312-d842-40fc-be8a-5ef3fb1eca79-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4\" (UID: \"1ba65312-d842-40fc-be8a-5ef3fb1eca79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"
Sep 16 12:28:28.793874 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:28.793838    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"
Sep 16 12:28:29.069491 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:29.069465    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"]
Sep 16 12:28:29.071052 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:28:29.071023    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba65312_d842_40fc_be8a_5ef3fb1eca79.slice/crio-b706b2e22ce95b1634448a2ce8b88ae171af180f0d6b353d8b326c27549d6d11 WatchSource:0}: Error finding container b706b2e22ce95b1634448a2ce8b88ae171af180f0d6b353d8b326c27549d6d11: Status 404 returned error can't find the container with id b706b2e22ce95b1634448a2ce8b88ae171af180f0d6b353d8b326c27549d6d11
Sep 16 12:28:29.902211 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:29.902174    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4" event={"ID":"1ba65312-d842-40fc-be8a-5ef3fb1eca79","Type":"ContainerStarted","Data":"a03346a09c7e8b76b6b24af0ce5f6f0d56862aba366487971f9acb596e036e96"}
Sep 16 12:28:29.902691 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:29.902226    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4" event={"ID":"1ba65312-d842-40fc-be8a-5ef3fb1eca79","Type":"ContainerStarted","Data":"b706b2e22ce95b1634448a2ce8b88ae171af180f0d6b353d8b326c27549d6d11"}
Sep 16 12:28:29.902691 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:29.902263    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"
Sep 16 12:28:29.921364 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:29.921313    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4" podStartSLOduration=3.583878593 podStartE2EDuration="3.921299609s" totalImagesPullingTime="337.421016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:28:29.918730645 +0000 UTC m=+1503.619801973" watchObservedRunningTime="2026-09-16 12:28:29.921299609 +0000 UTC m=+1503.622370935"
Sep 16 12:28:35.914165 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:35.914136    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"
Sep 16 12:28:38.823752 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:38.823717    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"]
Sep 16 12:28:38.824214 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:38.824196    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f35bc2fb-f32f-4f0f-8e78-918f927616e3" containerName="istio-proxy"
Sep 16 12:28:38.824288 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:38.824218    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f35bc2fb-f32f-4f0f-8e78-918f927616e3" containerName="istio-proxy"
Sep 16 12:28:38.824343 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:38.824334    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2316ba71-eb2c-48d2-a6e2-cb2bd6bdf2c1" containerName="istio-proxy"
Sep 16 12:28:38.824380 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:38.824348    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f35bc2fb-f32f-4f0f-8e78-918f927616e3" containerName="istio-proxy"
Sep 16 12:28:38.826733 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:38.826708    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:38.829307 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:38.829284    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-0e94e5-openshift-default-dockercfg-wjv8l\""
Sep 16 12:28:38.838803 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:38.838776    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"]
Sep 16 12:28:38.952591 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:38.952560    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istio-podinfo\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:38.952785 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:38.952606    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-workload-certs\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:38.952785 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:38.952719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istio-token\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:38.952785 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:38.952747    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6mgj\" (UniqueName: \"kubernetes.io/projected/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-kube-api-access-g6mgj\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:38.952785 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:38.952781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-workload-socket\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:38.952933 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:38.952885    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istiod-ca-cert\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:38.952933 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:38.952910    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istio-envoy\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:38.952997 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:38.952933    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-credential-socket\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:38.953029 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:38.953000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istio-data\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:39.054262 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.054227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-workload-socket\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:39.054439 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.054304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istiod-ca-cert\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:39.054439 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.054325    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istio-envoy\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:39.054439 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.054345    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-credential-socket\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:39.054439 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.054373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istio-data\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:39.054439 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.054389    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istio-podinfo\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:39.054712 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.054523    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-workload-certs\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:39.054712 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.054621    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istio-token\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:39.054712 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.054670    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-workload-socket\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:39.054862 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.054799    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istio-data\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:39.054862 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.054648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6mgj\" (UniqueName: \"kubernetes.io/projected/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-kube-api-access-g6mgj\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:39.054985 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.054959    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-workload-certs\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:39.055095 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.055075    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-credential-socket\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:39.055133 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.055117    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istiod-ca-cert\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:39.056621 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.056597    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istio-envoy\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:39.056864 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.056847    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istio-podinfo\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:39.062018 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.061998    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istio-token\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:39.062335 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.062315    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6mgj\" (UniqueName: \"kubernetes.io/projected/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-kube-api-access-g6mgj\") pod \"e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") " pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:39.139768 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.139728    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:39.418962 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.418930    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"]
Sep 16 12:28:39.419886 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:28:39.419861    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18e3d54_bb46_4c26_8c7d_b8b4881594b5.slice/crio-735880ea7ab2ab2f0d350b75638d7e178a7f81ec000029529ff0ebad7837aaf5 WatchSource:0}: Error finding container 735880ea7ab2ab2f0d350b75638d7e178a7f81ec000029529ff0ebad7837aaf5: Status 404 returned error can't find the container with id 735880ea7ab2ab2f0d350b75638d7e178a7f81ec000029529ff0ebad7837aaf5
Sep 16 12:28:39.422040 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.422004    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:28:39.422128 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.422086    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:28:39.422197 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.422134    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:28:39.961633 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.961586    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn" event={"ID":"a18e3d54-bb46-4c26-8c7d-b8b4881594b5","Type":"ContainerStarted","Data":"79d2e3fda5fd9c1fc2f4feacbbef2ed13b4dc123ab6868626da5d3b2004d0bf6"}
Sep 16 12:28:39.962112 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.961651    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn" event={"ID":"a18e3d54-bb46-4c26-8c7d-b8b4881594b5","Type":"ContainerStarted","Data":"735880ea7ab2ab2f0d350b75638d7e178a7f81ec000029529ff0ebad7837aaf5"}
Sep 16 12:28:39.986998 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:39.986946    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn" podStartSLOduration=1.986930973 podStartE2EDuration="1.986930973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:28:39.984199224 +0000 UTC m=+1513.685270550" watchObservedRunningTime="2026-09-16 12:28:39.986930973 +0000 UTC m=+1513.688002299"
Sep 16 12:28:40.140011 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:40.139976    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:40.145804 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:40.145779    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:40.968048 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:40.968005    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:40.969027 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:40.969005    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:28:46.213417 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:46.213379    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq"]
Sep 16 12:28:46.216190 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:46.216171    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq"
Sep 16 12:28:46.218715 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:46.218684    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-0e94e5-serving-cert\""
Sep 16 12:28:46.228035 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:46.228008    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq"]
Sep 16 12:28:46.332171 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:46.332138    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq\" (UID: \"cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq"
Sep 16 12:28:46.332320 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:46.332218    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4nd7\" (UniqueName: \"kubernetes.io/projected/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-kube-api-access-n4nd7\") pod \"maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq\" (UID: \"cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq"
Sep 16 12:28:46.332359 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:46.332318    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq\" (UID: \"cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq"
Sep 16 12:28:46.332359 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:46.332350    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq\" (UID: \"cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq"
Sep 16 12:28:46.433304 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:46.433265    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4nd7\" (UniqueName: \"kubernetes.io/projected/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-kube-api-access-n4nd7\") pod \"maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq\" (UID: \"cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq"
Sep 16 12:28:46.433455 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:46.433352    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq\" (UID: \"cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq"
Sep 16 12:28:46.433455 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:46.433375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq\" (UID: \"cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq"
Sep 16 12:28:46.433455 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:46.433406    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq\" (UID: \"cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq"
Sep 16 12:28:46.433634 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:28:46.433535    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-serving-cert: secret "maas-api-e2e-shared-b-w3-0e94e5-serving-cert" not found
Sep 16 12:28:46.433634 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:28:46.433617    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-maas-api-tls podName:cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d nodeName:}" failed. No retries permitted until 2026-09-16 12:28:46.933593713 +0000 UTC m=+1520.634665024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-maas-api-tls") pod "maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq" (UID: "cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d") : secret "maas-api-e2e-shared-b-w3-0e94e5-serving-cert" not found
Sep 16 12:28:46.434062 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:46.434042    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq\" (UID: \"cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq"
Sep 16 12:28:46.435838 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:46.435822    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq\" (UID: \"cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq"
Sep 16 12:28:46.444874 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:46.444848    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4nd7\" (UniqueName: \"kubernetes.io/projected/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-kube-api-access-n4nd7\") pod \"maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq\" (UID: \"cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq"
Sep 16 12:28:46.938034 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:46.937991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq\" (UID: \"cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq"
Sep 16 12:28:46.943049 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:46.942040    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq\" (UID: \"cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq"
Sep 16 12:28:47.129045 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:47.129009    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq"
Sep 16 12:28:47.406108 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:47.406080    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq"]
Sep 16 12:28:47.406925 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:28:47.406895    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde23e4a_dbc2_4b0c_90aa_4aa34ba14c8d.slice/crio-16eb846f91019322e28d431d48384f34701f62119eebcfa5f398a8313dd91454 WatchSource:0}: Error finding container 16eb846f91019322e28d431d48384f34701f62119eebcfa5f398a8313dd91454: Status 404 returned error can't find the container with id 16eb846f91019322e28d431d48384f34701f62119eebcfa5f398a8313dd91454
Sep 16 12:28:48.012267 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:48.012222    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq" event={"ID":"cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d","Type":"ContainerStarted","Data":"16eb846f91019322e28d431d48384f34701f62119eebcfa5f398a8313dd91454"}
Sep 16 12:28:49.019998 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:49.019915    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq" event={"ID":"cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d","Type":"ContainerStarted","Data":"baacecc163667942cfef1c6a348a2a3606daee83d6174739377b81854b7e5574"}
Sep 16 12:28:49.020362 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:49.020038    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq"
Sep 16 12:28:49.037168 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:49.037110    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq" podStartSLOduration=2.508255176 podStartE2EDuration="3.037094756s" totalImagesPullingTime="528.83958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:28:49.035963175 +0000 UTC m=+1522.737034501" watchObservedRunningTime="2026-09-16 12:28:49.037094756 +0000 UTC m=+1522.738166082"
Sep 16 12:28:50.380485 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:50.380452    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"]
Sep 16 12:28:50.381044 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:50.380725    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28" podUID="c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12" containerName="maas-api" containerID="cri-o://000476e771b3005d991e5f7a17f54c0d1831250bb48afbf00562ec356e9a802a" gracePeriod=30
Sep 16 12:28:50.631696 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:50.631613    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"
Sep 16 12:28:50.777132 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:50.777103    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-maas-api-tls\") pod \"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12\" (UID: \"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12\") "
Sep 16 12:28:50.777326 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:50.777193    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-metrics-certs\") pod \"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12\" (UID: \"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12\") "
Sep 16 12:28:50.777326 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:50.777247    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-openshift-service-ca\") pod \"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12\" (UID: \"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12\") "
Sep 16 12:28:50.777326 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:50.777274    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xkcn\" (UniqueName: \"kubernetes.io/projected/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-kube-api-access-7xkcn\") pod \"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12\" (UID: \"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12\") "
Sep 16 12:28:50.777907 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:50.777842    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12" (UID: "c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:28:50.801678 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:50.784234    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-openshift-service-ca\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:28:50.801678 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:50.790778    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-kube-api-access-7xkcn" (OuterVolumeSpecName: "kube-api-access-7xkcn") pod "c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12" (UID: "c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12"). InnerVolumeSpecName "kube-api-access-7xkcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:28:50.801678 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:50.800345    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12" (UID: "c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:28:50.814744 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:50.812545    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12" (UID: "c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:28:50.885518 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:50.885431    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xkcn\" (UniqueName: \"kubernetes.io/projected/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-kube-api-access-7xkcn\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:28:50.885518 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:50.885463    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-maas-api-tls\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:28:50.885518 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:50.885473    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12-metrics-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:28:51.033175 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:51.033128    2579 generic.go:356] "Generic (PLEG): container finished" podID="c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12" containerID="000476e771b3005d991e5f7a17f54c0d1831250bb48afbf00562ec356e9a802a" exitCode=0
Sep 16 12:28:51.033175 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:51.033187    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"
Sep 16 12:28:51.033396 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:51.033218    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28" event={"ID":"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12","Type":"ContainerDied","Data":"000476e771b3005d991e5f7a17f54c0d1831250bb48afbf00562ec356e9a802a"}
Sep 16 12:28:51.033396 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:51.033278    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28" event={"ID":"c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12","Type":"ContainerDied","Data":"51caafc08ebeddd928aaa865848f37fa5ecf46c04e7a98ac30baac99435bf0c3"}
Sep 16 12:28:51.033396 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:51.033302    2579 scope.go:117] "RemoveContainer" containerID="000476e771b3005d991e5f7a17f54c0d1831250bb48afbf00562ec356e9a802a"
Sep 16 12:28:51.046443 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:51.046420    2579 scope.go:117] "RemoveContainer" containerID="000476e771b3005d991e5f7a17f54c0d1831250bb48afbf00562ec356e9a802a"
Sep 16 12:28:51.046758 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:28:51.046736    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"000476e771b3005d991e5f7a17f54c0d1831250bb48afbf00562ec356e9a802a\": container with ID starting with 000476e771b3005d991e5f7a17f54c0d1831250bb48afbf00562ec356e9a802a not found: ID does not exist" containerID="000476e771b3005d991e5f7a17f54c0d1831250bb48afbf00562ec356e9a802a"
Sep 16 12:28:51.046839 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:51.046766    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"000476e771b3005d991e5f7a17f54c0d1831250bb48afbf00562ec356e9a802a"} err="failed to get container status \"000476e771b3005d991e5f7a17f54c0d1831250bb48afbf00562ec356e9a802a\": rpc error: code = NotFound desc = could not find container \"000476e771b3005d991e5f7a17f54c0d1831250bb48afbf00562ec356e9a802a\": container with ID starting with 000476e771b3005d991e5f7a17f54c0d1831250bb48afbf00562ec356e9a802a not found: ID does not exist"
Sep 16 12:28:51.057589 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:51.057551    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"]
Sep 16 12:28:51.060626 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:51.060601    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5891124-7d95c945bd-wgd28"]
Sep 16 12:28:51.171212 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:51.171122    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:28:52.783673 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:52.783622    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12" path="/var/lib/kubelet/pods/c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12/volumes"
Sep 16 12:28:55.031291 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:28:55.031258    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq"
Sep 16 12:29:18.227453 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:29:18.227417    2579 generic.go:356] "Generic (PLEG): container finished" podID="a18e3d54-bb46-4c26-8c7d-b8b4881594b5" containerID="79d2e3fda5fd9c1fc2f4feacbbef2ed13b4dc123ab6868626da5d3b2004d0bf6" exitCode=0
Sep 16 12:29:18.227897 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:29:18.227489    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn" event={"ID":"a18e3d54-bb46-4c26-8c7d-b8b4881594b5","Type":"ContainerDied","Data":"79d2e3fda5fd9c1fc2f4feacbbef2ed13b4dc123ab6868626da5d3b2004d0bf6"}
Sep 16 12:29:18.227998 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:29:18.227973    2579 scope.go:117] "RemoveContainer" containerID="79d2e3fda5fd9c1fc2f4feacbbef2ed13b4dc123ab6868626da5d3b2004d0bf6"
Sep 16 12:29:18.229342 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:29:18.229314    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:29:18.229418 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:29:18.229385    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:29:18.229457 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:29:18.229417    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:29:19.235540 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:29:19.235507    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn" event={"ID":"a18e3d54-bb46-4c26-8c7d-b8b4881594b5","Type":"ContainerStarted","Data":"d6864e4a50ba63f9aa9e68f44c093c340f97f60783d2d93e646ced19091d02ab"}
Sep 16 12:29:20.140235 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:29:20.140204    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:29:20.140511 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:29:20.140487    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 16 12:29:20.140557 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:29:20.140537    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn" podUID="a18e3d54-bb46-4c26-8c7d-b8b4881594b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 16 12:29:20.766895 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:29:20.766856    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:29:21.140599 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:29:21.140564    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 16 12:29:21.140795 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:29:21.140626    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn" podUID="a18e3d54-bb46-4c26-8c7d-b8b4881594b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 16 12:29:22.140642 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:29:22.140603    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 16 12:29:22.141040 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:29:22.140697    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn" podUID="a18e3d54-bb46-4c26-8c7d-b8b4881594b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 16 12:29:23.141031 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:29:23.140988    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 16 12:29:23.141463 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:29:23.141053    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn" podUID="a18e3d54-bb46-4c26-8c7d-b8b4881594b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 16 12:29:24.140986 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:29:24.140943    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 16 12:29:24.141173 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:29:24.141004    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn" podUID="a18e3d54-bb46-4c26-8c7d-b8b4881594b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 16 12:29:25.144980 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:29:25.144945    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:29:25.145490 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:29:25.145339    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:29:25.145965 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:29:25.145943    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:29:47.773671 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:29:47.773619    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:30:00.166170 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:00.166131    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-739ee7-29826030-zpswx"]
Sep 16 12:30:00.166709 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:00.166561    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12" containerName="maas-api"
Sep 16 12:30:00.166709 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:00.166572    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12" containerName="maas-api"
Sep 16 12:30:00.166709 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:00.166677    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c9272c-a6fb-47f4-bed8-9e13c2e1ae12" containerName="maas-api"
Sep 16 12:30:00.169310 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:00.169287    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-739ee7-29826030-zpswx"
Sep 16 12:30:00.174133 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:00.174107    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-767vd\""
Sep 16 12:30:00.181215 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:00.181191    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-739ee7-29826030-zpswx"
Sep 16 12:30:00.181564 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:00.181541    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-739ee7-29826030-zpswx"]
Sep 16 12:30:00.460447 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:00.460416    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-739ee7-29826030-zpswx"]
Sep 16 12:30:00.461968 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:30:00.461944    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e4a997_eacd_49df_8c6e_4298c01b640b.slice/crio-e5a2311e29520c0309fa2620b25197ac5bbe234c32c75c171b8c281d4ed63b41 WatchSource:0}: Error finding container e5a2311e29520c0309fa2620b25197ac5bbe234c32c75c171b8c281d4ed63b41: Status 404 returned error can't find the container with id e5a2311e29520c0309fa2620b25197ac5bbe234c32c75c171b8c281d4ed63b41
Sep 16 12:30:00.497564 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:00.497531    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-739ee7-29826030-zpswx" event={"ID":"67e4a997-eacd-49df-8c6e-4298c01b640b","Type":"ContainerStarted","Data":"e5a2311e29520c0309fa2620b25197ac5bbe234c32c75c171b8c281d4ed63b41"}
Sep 16 12:30:01.505010 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:01.504971    2579 generic.go:356] "Generic (PLEG): container finished" podID="67e4a997-eacd-49df-8c6e-4298c01b640b" containerID="c9f2996b4cc05ede55a05135b31bb8a791300f7f186659ed7bbfeaea828ea201" exitCode=0
Sep 16 12:30:01.505010 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:01.505004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-739ee7-29826030-zpswx" event={"ID":"67e4a997-eacd-49df-8c6e-4298c01b640b","Type":"ContainerDied","Data":"c9f2996b4cc05ede55a05135b31bb8a791300f7f186659ed7bbfeaea828ea201"}
Sep 16 12:30:02.651075 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:02.651047    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-739ee7-29826030-zpswx"
Sep 16 12:30:03.518356 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:03.518327    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-739ee7-29826030-zpswx"
Sep 16 12:30:03.518569 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:03.518362    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-739ee7-29826030-zpswx" event={"ID":"67e4a997-eacd-49df-8c6e-4298c01b640b","Type":"ContainerDied","Data":"e5a2311e29520c0309fa2620b25197ac5bbe234c32c75c171b8c281d4ed63b41"}
Sep 16 12:30:03.518569 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:03.518410    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5a2311e29520c0309fa2620b25197ac5bbe234c32c75c171b8c281d4ed63b41"
Sep 16 12:30:05.188387 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:05.188354    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"]
Sep 16 12:30:05.188787 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:05.188569    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9" podUID="30e0588a-f0b0-425c-9c5c-cb41c5696981" containerName="istio-proxy" containerID="cri-o://5de4efdf7fcb442d532747986e8f2b27514a5a645a2ddd2edfcee45625d0f65c" gracePeriod=30
Sep 16 12:30:10.441567 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.441541    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:30:10.499967 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.499888    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-istio-envoy\") pod \"30e0588a-f0b0-425c-9c5c-cb41c5696981\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") "
Sep 16 12:30:10.499967 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.499936    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btdb2\" (UniqueName: \"kubernetes.io/projected/30e0588a-f0b0-425c-9c5c-cb41c5696981-kube-api-access-btdb2\") pod \"30e0588a-f0b0-425c-9c5c-cb41c5696981\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") "
Sep 16 12:30:10.499967 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.499954    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-credential-socket\") pod \"30e0588a-f0b0-425c-9c5c-cb41c5696981\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") "
Sep 16 12:30:10.500199 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.499974    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-istio-data\") pod \"30e0588a-f0b0-425c-9c5c-cb41c5696981\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") "
Sep 16 12:30:10.500199 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.500034    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30e0588a-f0b0-425c-9c5c-cb41c5696981-istio-podinfo\") pod \"30e0588a-f0b0-425c-9c5c-cb41c5696981\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") "
Sep 16 12:30:10.500199 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.500072    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30e0588a-f0b0-425c-9c5c-cb41c5696981-istiod-ca-cert\") pod \"30e0588a-f0b0-425c-9c5c-cb41c5696981\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") "
Sep 16 12:30:10.500199 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.500106    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30e0588a-f0b0-425c-9c5c-cb41c5696981-istio-token\") pod \"30e0588a-f0b0-425c-9c5c-cb41c5696981\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") "
Sep 16 12:30:10.500199 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.500131    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-workload-certs\") pod \"30e0588a-f0b0-425c-9c5c-cb41c5696981\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") "
Sep 16 12:30:10.500199 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.500158    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-workload-socket\") pod \"30e0588a-f0b0-425c-9c5c-cb41c5696981\" (UID: \"30e0588a-f0b0-425c-9c5c-cb41c5696981\") "
Sep 16 12:30:10.500536 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.500211    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "30e0588a-f0b0-425c-9c5c-cb41c5696981" (UID: "30e0588a-f0b0-425c-9c5c-cb41c5696981"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:30:10.500536 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.500482    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "30e0588a-f0b0-425c-9c5c-cb41c5696981" (UID: "30e0588a-f0b0-425c-9c5c-cb41c5696981"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:30:10.500668 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.500551    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30e0588a-f0b0-425c-9c5c-cb41c5696981-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "30e0588a-f0b0-425c-9c5c-cb41c5696981" (UID: "30e0588a-f0b0-425c-9c5c-cb41c5696981"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:30:10.500668 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.500569    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-credential-socket\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:30:10.500668 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.500589    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-workload-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:30:10.500824 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.500709    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "30e0588a-f0b0-425c-9c5c-cb41c5696981" (UID: "30e0588a-f0b0-425c-9c5c-cb41c5696981"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:30:10.500891 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.500840    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-istio-data" (OuterVolumeSpecName: "istio-data") pod "30e0588a-f0b0-425c-9c5c-cb41c5696981" (UID: "30e0588a-f0b0-425c-9c5c-cb41c5696981"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:30:10.502173 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.502149    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "30e0588a-f0b0-425c-9c5c-cb41c5696981" (UID: "30e0588a-f0b0-425c-9c5c-cb41c5696981"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:30:10.502299 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.502254    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30e0588a-f0b0-425c-9c5c-cb41c5696981-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "30e0588a-f0b0-425c-9c5c-cb41c5696981" (UID: "30e0588a-f0b0-425c-9c5c-cb41c5696981"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:30:10.502299 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.502279    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e0588a-f0b0-425c-9c5c-cb41c5696981-istio-token" (OuterVolumeSpecName: "istio-token") pod "30e0588a-f0b0-425c-9c5c-cb41c5696981" (UID: "30e0588a-f0b0-425c-9c5c-cb41c5696981"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:30:10.502299 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.502292    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e0588a-f0b0-425c-9c5c-cb41c5696981-kube-api-access-btdb2" (OuterVolumeSpecName: "kube-api-access-btdb2") pod "30e0588a-f0b0-425c-9c5c-cb41c5696981" (UID: "30e0588a-f0b0-425c-9c5c-cb41c5696981"). InnerVolumeSpecName "kube-api-access-btdb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:30:10.561206 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.561173    2579 generic.go:356] "Generic (PLEG): container finished" podID="30e0588a-f0b0-425c-9c5c-cb41c5696981" containerID="5de4efdf7fcb442d532747986e8f2b27514a5a645a2ddd2edfcee45625d0f65c" exitCode=0
Sep 16 12:30:10.561353 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.561225    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"
Sep 16 12:30:10.561353 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.561247    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9" event={"ID":"30e0588a-f0b0-425c-9c5c-cb41c5696981","Type":"ContainerDied","Data":"5de4efdf7fcb442d532747986e8f2b27514a5a645a2ddd2edfcee45625d0f65c"}
Sep 16 12:30:10.561353 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.561302    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9" event={"ID":"30e0588a-f0b0-425c-9c5c-cb41c5696981","Type":"ContainerDied","Data":"3fdb89dbe2742426bc316f0e5a50389f742b54a254e3e8e72a63d255a575c19d"}
Sep 16 12:30:10.561353 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.561324    2579 scope.go:117] "RemoveContainer" containerID="5de4efdf7fcb442d532747986e8f2b27514a5a645a2ddd2edfcee45625d0f65c"
Sep 16 12:30:10.572069 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.572053    2579 scope.go:117] "RemoveContainer" containerID="04d6e38555c514b90677995c2c4729d07000644caeb3a6cedda234466e7d7335"
Sep 16 12:30:10.583929 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.583420    2579 scope.go:117] "RemoveContainer" containerID="5de4efdf7fcb442d532747986e8f2b27514a5a645a2ddd2edfcee45625d0f65c"
Sep 16 12:30:10.584462 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:30:10.584426    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de4efdf7fcb442d532747986e8f2b27514a5a645a2ddd2edfcee45625d0f65c\": container with ID starting with 5de4efdf7fcb442d532747986e8f2b27514a5a645a2ddd2edfcee45625d0f65c not found: ID does not exist" containerID="5de4efdf7fcb442d532747986e8f2b27514a5a645a2ddd2edfcee45625d0f65c"
Sep 16 12:30:10.584572 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.584473    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de4efdf7fcb442d532747986e8f2b27514a5a645a2ddd2edfcee45625d0f65c"} err="failed to get container status \"5de4efdf7fcb442d532747986e8f2b27514a5a645a2ddd2edfcee45625d0f65c\": rpc error: code = NotFound desc = could not find container \"5de4efdf7fcb442d532747986e8f2b27514a5a645a2ddd2edfcee45625d0f65c\": container with ID starting with 5de4efdf7fcb442d532747986e8f2b27514a5a645a2ddd2edfcee45625d0f65c not found: ID does not exist"
Sep 16 12:30:10.584572 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.584502    2579 scope.go:117] "RemoveContainer" containerID="04d6e38555c514b90677995c2c4729d07000644caeb3a6cedda234466e7d7335"
Sep 16 12:30:10.584863 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:30:10.584836    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d6e38555c514b90677995c2c4729d07000644caeb3a6cedda234466e7d7335\": container with ID starting with 04d6e38555c514b90677995c2c4729d07000644caeb3a6cedda234466e7d7335 not found: ID does not exist" containerID="04d6e38555c514b90677995c2c4729d07000644caeb3a6cedda234466e7d7335"
Sep 16 12:30:10.584929 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.584870    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d6e38555c514b90677995c2c4729d07000644caeb3a6cedda234466e7d7335"} err="failed to get container status \"04d6e38555c514b90677995c2c4729d07000644caeb3a6cedda234466e7d7335\": rpc error: code = NotFound desc = could not find container \"04d6e38555c514b90677995c2c4729d07000644caeb3a6cedda234466e7d7335\": container with ID starting with 04d6e38555c514b90677995c2c4729d07000644caeb3a6cedda234466e7d7335 not found: ID does not exist"
Sep 16 12:30:10.587839 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.587819    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"]
Sep 16 12:30:10.590404 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.590386    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d5891124-openshift-default-7bb7875776-mxjt9"]
Sep 16 12:30:10.601619 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.601596    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30e0588a-f0b0-425c-9c5c-cb41c5696981-istio-podinfo\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:30:10.601619 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.601615    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30e0588a-f0b0-425c-9c5c-cb41c5696981-istiod-ca-cert\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:30:10.601755 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.601624    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30e0588a-f0b0-425c-9c5c-cb41c5696981-istio-token\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:30:10.601755 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.601632    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-workload-socket\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:30:10.601755 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.601640    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-istio-envoy\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:30:10.601755 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.601649    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btdb2\" (UniqueName: \"kubernetes.io/projected/30e0588a-f0b0-425c-9c5c-cb41c5696981-kube-api-access-btdb2\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:30:10.601755 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.601673    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30e0588a-f0b0-425c-9c5c-cb41c5696981-istio-data\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:30:10.783399 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:10.783323    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30e0588a-f0b0-425c-9c5c-cb41c5696981" path="/var/lib/kubelet/pods/30e0588a-f0b0-425c-9c5c-cb41c5696981/volumes"
Sep 16 12:30:18.710714 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:18.710674    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:30:37.970250 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:37.970214    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"]
Sep 16 12:30:37.970781 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:37.970761    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e4a997-eacd-49df-8c6e-4298c01b640b" containerName="cleanup"
Sep 16 12:30:37.970858 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:37.970785    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e4a997-eacd-49df-8c6e-4298c01b640b" containerName="cleanup"
Sep 16 12:30:37.970858 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:37.970803    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30e0588a-f0b0-425c-9c5c-cb41c5696981" containerName="istio-proxy"
Sep 16 12:30:37.970858 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:37.970811    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="30e0588a-f0b0-425c-9c5c-cb41c5696981" containerName="istio-proxy"
Sep 16 12:30:37.970858 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:37.970827    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30e0588a-f0b0-425c-9c5c-cb41c5696981" containerName="istio-proxy"
Sep 16 12:30:37.970858 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:37.970836    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="30e0588a-f0b0-425c-9c5c-cb41c5696981" containerName="istio-proxy"
Sep 16 12:30:37.971035 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:37.970942    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="30e0588a-f0b0-425c-9c5c-cb41c5696981" containerName="istio-proxy"
Sep 16 12:30:37.971035 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:37.970952    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="30e0588a-f0b0-425c-9c5c-cb41c5696981" containerName="istio-proxy"
Sep 16 12:30:37.971035 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:37.970963    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e4a997-eacd-49df-8c6e-4298c01b640b" containerName="cleanup"
Sep 16 12:30:37.974165 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:37.974149    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:30:37.977383 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:37.977364    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-739ee7\"/\"embedding-w0-739ee7-kserve-self-signed-certs\""
Sep 16 12:30:37.988473 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:37.988448    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"]
Sep 16 12:30:38.084641 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:38.084607    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-tmp-dir\") pod \"embedding-w0-739ee7-kserve-74b488956d-9rfsc\" (UID: \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\") " pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:30:38.084641 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:38.084648    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-model-cache\") pod \"embedding-w0-739ee7-kserve-74b488956d-9rfsc\" (UID: \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\") " pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:30:38.084949 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:38.084735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tq7j\" (UniqueName: \"kubernetes.io/projected/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-kube-api-access-4tq7j\") pod \"embedding-w0-739ee7-kserve-74b488956d-9rfsc\" (UID: \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\") " pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:30:38.084949 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:38.084769    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-dshm\") pod \"embedding-w0-739ee7-kserve-74b488956d-9rfsc\" (UID: \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\") " pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:30:38.084949 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:38.084837    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-home\") pod \"embedding-w0-739ee7-kserve-74b488956d-9rfsc\" (UID: \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\") " pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:30:38.084949 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:38.084881    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-tls-certs\") pod \"embedding-w0-739ee7-kserve-74b488956d-9rfsc\" (UID: \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\") " pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:30:38.185986 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:38.185945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-tmp-dir\") pod \"embedding-w0-739ee7-kserve-74b488956d-9rfsc\" (UID: \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\") " pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:30:38.185986 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:38.185991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-model-cache\") pod \"embedding-w0-739ee7-kserve-74b488956d-9rfsc\" (UID: \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\") " pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:30:38.186243 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:38.186010    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tq7j\" (UniqueName: \"kubernetes.io/projected/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-kube-api-access-4tq7j\") pod \"embedding-w0-739ee7-kserve-74b488956d-9rfsc\" (UID: \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\") " pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:30:38.186243 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:38.186033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-dshm\") pod \"embedding-w0-739ee7-kserve-74b488956d-9rfsc\" (UID: \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\") " pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:30:38.186243 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:38.186118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-home\") pod \"embedding-w0-739ee7-kserve-74b488956d-9rfsc\" (UID: \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\") " pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:30:38.186243 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:38.186174    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-tls-certs\") pod \"embedding-w0-739ee7-kserve-74b488956d-9rfsc\" (UID: \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\") " pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:30:38.186455 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:38.186429    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-model-cache\") pod \"embedding-w0-739ee7-kserve-74b488956d-9rfsc\" (UID: \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\") " pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:30:38.186512 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:38.186451    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-tmp-dir\") pod \"embedding-w0-739ee7-kserve-74b488956d-9rfsc\" (UID: \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\") " pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:30:38.186512 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:38.186496    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-home\") pod \"embedding-w0-739ee7-kserve-74b488956d-9rfsc\" (UID: \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\") " pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:30:38.188365 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:38.188338    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-dshm\") pod \"embedding-w0-739ee7-kserve-74b488956d-9rfsc\" (UID: \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\") " pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:30:38.188610 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:38.188594    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-tls-certs\") pod \"embedding-w0-739ee7-kserve-74b488956d-9rfsc\" (UID: \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\") " pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:30:38.197873 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:38.197848    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tq7j\" (UniqueName: \"kubernetes.io/projected/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-kube-api-access-4tq7j\") pod \"embedding-w0-739ee7-kserve-74b488956d-9rfsc\" (UID: \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\") " pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:30:38.283024 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:38.282934    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:30:38.591430 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:38.591392    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"]
Sep 16 12:30:38.591741 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:30:38.591715    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb97355ae_45e5_4812_b0b1_3abc6d7a0c69.slice/crio-b8bd1f209731c09988d897ff0721eec9f6c840fe62c180960393c2bc8b0ff7cf WatchSource:0}: Error finding container b8bd1f209731c09988d897ff0721eec9f6c840fe62c180960393c2bc8b0ff7cf: Status 404 returned error can't find the container with id b8bd1f209731c09988d897ff0721eec9f6c840fe62c180960393c2bc8b0ff7cf
Sep 16 12:30:38.734680 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:38.734615    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc" event={"ID":"b97355ae-45e5-4812-b0b1-3abc6d7a0c69","Type":"ContainerStarted","Data":"6caf0bd52ba04f49cb8d6532eb529d31caa7d3c76d97985ddee1145fffad816b"}
Sep 16 12:30:38.734840 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:38.734686    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc" event={"ID":"b97355ae-45e5-4812-b0b1-3abc6d7a0c69","Type":"ContainerStarted","Data":"b8bd1f209731c09988d897ff0721eec9f6c840fe62c180960393c2bc8b0ff7cf"}
Sep 16 12:30:38.760582 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:38.760531    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc" podStartSLOduration=1.760514896 podStartE2EDuration="1.760514896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:30:38.759787514 +0000 UTC m=+1632.460858841" watchObservedRunningTime="2026-09-16 12:30:38.760514896 +0000 UTC m=+1632.461586222"
Sep 16 12:30:48.283773 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:48.283732    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:30:48.284264 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:48.283795    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:30:48.297227 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:48.297199    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:30:48.808125 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:48.808092    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:30:52.689234 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:30:52.689190    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:31:01.483217 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.483178    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"]
Sep 16 12:31:01.486898 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.486873    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.491048 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.491023    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-8641cd-openshift-default-dockercfg-th8v7\""
Sep 16 12:31:01.511596 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.511561    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"]
Sep 16 12:31:01.613388 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.613348    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-655sj\" (UniqueName: \"kubernetes.io/projected/eaa9624f-72d7-4a15-abcc-ba19f761388a-kube-api-access-655sj\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.613580 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.613393    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-istio-envoy\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.613580 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.613448    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-workload-certs\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.613580 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.613511    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa9624f-72d7-4a15-abcc-ba19f761388a-istio-podinfo\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.613580 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.613559    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-istio-data\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.613580 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.613580    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa9624f-72d7-4a15-abcc-ba19f761388a-istio-token\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.613838 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.613611    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-workload-socket\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.613838 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.613749    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa9624f-72d7-4a15-abcc-ba19f761388a-istiod-ca-cert\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.613838 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.613814    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-credential-socket\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.715313 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.715267    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa9624f-72d7-4a15-abcc-ba19f761388a-istiod-ca-cert\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.715313 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.715316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-credential-socket\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.715567 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.715337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-655sj\" (UniqueName: \"kubernetes.io/projected/eaa9624f-72d7-4a15-abcc-ba19f761388a-kube-api-access-655sj\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.715567 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.715355    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-istio-envoy\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.715567 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.715374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-workload-certs\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.715567 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.715406    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa9624f-72d7-4a15-abcc-ba19f761388a-istio-podinfo\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.715567 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.715444    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-istio-data\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.715567 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.715465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa9624f-72d7-4a15-abcc-ba19f761388a-istio-token\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.715567 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.715505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-workload-socket\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.715961 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.715813    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-credential-socket\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.715961 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.715838    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-workload-certs\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.715961 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.715917    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-workload-socket\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.716134 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.716053    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-istio-data\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.716244 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.716221    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa9624f-72d7-4a15-abcc-ba19f761388a-istiod-ca-cert\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.717786 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.717765    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-istio-envoy\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.718004 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.717986    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa9624f-72d7-4a15-abcc-ba19f761388a-istio-podinfo\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.730293 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.730264    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa9624f-72d7-4a15-abcc-ba19f761388a-istio-token\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.730388 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.730343    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-655sj\" (UniqueName: \"kubernetes.io/projected/eaa9624f-72d7-4a15-abcc-ba19f761388a-kube-api-access-655sj\") pod \"e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") " pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:01.797866 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:01.797789    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:02.091012 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:02.090988    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"]
Sep 16 12:31:02.093041 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:31:02.093019    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa9624f_72d7_4a15_abcc_ba19f761388a.slice/crio-874c6907188e0746ba4240c6326510506368a15971816ff837cf394bc8967e19 WatchSource:0}: Error finding container 874c6907188e0746ba4240c6326510506368a15971816ff837cf394bc8967e19: Status 404 returned error can't find the container with id 874c6907188e0746ba4240c6326510506368a15971816ff837cf394bc8967e19
Sep 16 12:31:02.095282 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:02.095250    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:31:02.095349 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:02.095315    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:31:02.095349 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:02.095341    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:31:02.884420 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:02.884387    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5" event={"ID":"eaa9624f-72d7-4a15-abcc-ba19f761388a","Type":"ContainerStarted","Data":"7095f064a0dcda6d97873121788f0dcbc9ac21b08f71f09772e25cf45e21aaa9"}
Sep 16 12:31:02.884816 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:02.884437    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5" event={"ID":"eaa9624f-72d7-4a15-abcc-ba19f761388a","Type":"ContainerStarted","Data":"874c6907188e0746ba4240c6326510506368a15971816ff837cf394bc8967e19"}
Sep 16 12:31:02.922829 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:02.922776    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5" podStartSLOduration=1.922757654 podStartE2EDuration="1.922757654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:31:02.920075238 +0000 UTC m=+1656.621146564" watchObservedRunningTime="2026-09-16 12:31:02.922757654 +0000 UTC m=+1656.623828979"
Sep 16 12:31:03.798548 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:03.798512    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:03.803797 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:03.803768    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:03.891124 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:03.891092    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:03.892090 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:03.892073    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:14.192605 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:14.192571    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r"]
Sep 16 12:31:14.196619 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:14.196596    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r"
Sep 16 12:31:14.205175 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:31:14.205147    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r\" is forbidden: User \"system:node:ip-10-0-137-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-137-30.ec2.internal' and this object" podUID="b8cad60f-d2a0-4b30-a107-68f0bf623ae2" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r"
Sep 16 12:31:14.205260 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:31:14.205182    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-api-e2e-api-a-8641cd-serving-cert\" is forbidden: User \"system:node:ip-10-0-137-30.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-137-30.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-8641cd-serving-cert\"" type="*v1.Secret"
Sep 16 12:31:14.230182 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:14.230158    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r"]
Sep 16 12:31:14.353465 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:14.353426    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8cad60f-d2a0-4b30-a107-68f0bf623ae2-openshift-service-ca\") pod \"maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r\" (UID: \"b8cad60f-d2a0-4b30-a107-68f0bf623ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r"
Sep 16 12:31:14.353688 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:14.353477    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc7t7\" (UniqueName: \"kubernetes.io/projected/b8cad60f-d2a0-4b30-a107-68f0bf623ae2-kube-api-access-vc7t7\") pod \"maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r\" (UID: \"b8cad60f-d2a0-4b30-a107-68f0bf623ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r"
Sep 16 12:31:14.353688 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:14.353545    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8cad60f-d2a0-4b30-a107-68f0bf623ae2-metrics-certs\") pod \"maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r\" (UID: \"b8cad60f-d2a0-4b30-a107-68f0bf623ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r"
Sep 16 12:31:14.353688 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:14.353612    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8cad60f-d2a0-4b30-a107-68f0bf623ae2-maas-api-tls\") pod \"maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r\" (UID: \"b8cad60f-d2a0-4b30-a107-68f0bf623ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r"
Sep 16 12:31:14.454567 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:14.454477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8cad60f-d2a0-4b30-a107-68f0bf623ae2-openshift-service-ca\") pod \"maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r\" (UID: \"b8cad60f-d2a0-4b30-a107-68f0bf623ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r"
Sep 16 12:31:14.454567 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:14.454525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc7t7\" (UniqueName: \"kubernetes.io/projected/b8cad60f-d2a0-4b30-a107-68f0bf623ae2-kube-api-access-vc7t7\") pod \"maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r\" (UID: \"b8cad60f-d2a0-4b30-a107-68f0bf623ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r"
Sep 16 12:31:14.454567 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:14.454559    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8cad60f-d2a0-4b30-a107-68f0bf623ae2-metrics-certs\") pod \"maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r\" (UID: \"b8cad60f-d2a0-4b30-a107-68f0bf623ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r"
Sep 16 12:31:14.454824 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:14.454608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8cad60f-d2a0-4b30-a107-68f0bf623ae2-maas-api-tls\") pod \"maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r\" (UID: \"b8cad60f-d2a0-4b30-a107-68f0bf623ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r"
Sep 16 12:31:14.455225 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:14.455204    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8cad60f-d2a0-4b30-a107-68f0bf623ae2-openshift-service-ca\") pod \"maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r\" (UID: \"b8cad60f-d2a0-4b30-a107-68f0bf623ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r"
Sep 16 12:31:14.456994 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:14.456975    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8cad60f-d2a0-4b30-a107-68f0bf623ae2-metrics-certs\") pod \"maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r\" (UID: \"b8cad60f-d2a0-4b30-a107-68f0bf623ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r"
Sep 16 12:31:14.479514 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:14.479480    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc7t7\" (UniqueName: \"kubernetes.io/projected/b8cad60f-d2a0-4b30-a107-68f0bf623ae2-kube-api-access-vc7t7\") pod \"maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r\" (UID: \"b8cad60f-d2a0-4b30-a107-68f0bf623ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r"
Sep 16 12:31:15.128424 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:15.128391    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-8641cd-serving-cert\""
Sep 16 12:31:15.137790 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:15.137761    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8cad60f-d2a0-4b30-a107-68f0bf623ae2-maas-api-tls\") pod \"maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r\" (UID: \"b8cad60f-d2a0-4b30-a107-68f0bf623ae2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r"
Sep 16 12:31:15.407348 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:15.407246    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r"
Sep 16 12:31:15.726699 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:15.726595    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r"]
Sep 16 12:31:15.732893 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:31:15.732860    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8cad60f_d2a0_4b30_a107_68f0bf623ae2.slice/crio-dbc8ef9bf5284726828db6edbbeea3322662933a6a50706f431cc992e758c6f8 WatchSource:0}: Error finding container dbc8ef9bf5284726828db6edbbeea3322662933a6a50706f431cc992e758c6f8: Status 404 returned error can't find the container with id dbc8ef9bf5284726828db6edbbeea3322662933a6a50706f431cc992e758c6f8
Sep 16 12:31:15.734453 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:15.734431    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:31:15.968772 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:15.968725    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r" event={"ID":"b8cad60f-d2a0-4b30-a107-68f0bf623ae2","Type":"ContainerStarted","Data":"dbc8ef9bf5284726828db6edbbeea3322662933a6a50706f431cc992e758c6f8"}
Sep 16 12:31:16.976072 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:16.976034    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r" event={"ID":"b8cad60f-d2a0-4b30-a107-68f0bf623ae2","Type":"ContainerStarted","Data":"30f374b63531767015302049e81e0311f0e108a27ea3cfb095d90a84321f4754"}
Sep 16 12:31:16.976481 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:16.976100    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r"
Sep 16 12:31:22.989409 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:22.989379    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r"
Sep 16 12:31:23.015536 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:23.015490    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r" podStartSLOduration=8.586367196 podStartE2EDuration="9.015474018s" totalImagesPullingTime="429.106822ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:31:17.003436757 +0000 UTC m=+1670.704508095" watchObservedRunningTime="2026-09-16 12:31:23.015474018 +0000 UTC m=+1676.716545343"
Sep 16 12:31:31.171526 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:31.171475    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:31:43.153088 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:43.153054    2579 generic.go:356] "Generic (PLEG): container finished" podID="eaa9624f-72d7-4a15-abcc-ba19f761388a" containerID="7095f064a0dcda6d97873121788f0dcbc9ac21b08f71f09772e25cf45e21aaa9" exitCode=0
Sep 16 12:31:43.153545 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:43.153126    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5" event={"ID":"eaa9624f-72d7-4a15-abcc-ba19f761388a","Type":"ContainerDied","Data":"7095f064a0dcda6d97873121788f0dcbc9ac21b08f71f09772e25cf45e21aaa9"}
Sep 16 12:31:43.153594 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:43.153581    2579 scope.go:117] "RemoveContainer" containerID="7095f064a0dcda6d97873121788f0dcbc9ac21b08f71f09772e25cf45e21aaa9"
Sep 16 12:31:43.154840 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:43.154809    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:31:43.154954 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:43.154896    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:31:43.154954 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:43.154943    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 16 12:31:43.768756 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:43.768715    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"]
Sep 16 12:31:43.804177 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:43.804143    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"]
Sep 16 12:31:44.161229 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:44.161195    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5" event={"ID":"eaa9624f-72d7-4a15-abcc-ba19f761388a","Type":"ContainerStarted","Data":"f5c4dc5189fb059dae6beb582b75b8f388bbb40aaa0bb6e4cf42b1723a03aefc"}
Sep 16 12:31:44.798907 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:44.798872    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:44.800172 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:44.800145    2579 patch_prober.go:28] interesting pod/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 16 12:31:44.800299 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:44.800194    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5" podUID="eaa9624f-72d7-4a15-abcc-ba19f761388a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 16 12:31:45.798729 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:45.798679    2579 patch_prober.go:28] interesting pod/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 16 12:31:45.799254 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:45.798754    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5" podUID="eaa9624f-72d7-4a15-abcc-ba19f761388a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 16 12:31:46.799113 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:46.799071    2579 patch_prober.go:28] interesting pod/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 16 12:31:46.799588 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:46.799137    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5" podUID="eaa9624f-72d7-4a15-abcc-ba19f761388a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 16 12:31:47.798573 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:47.798529    2579 patch_prober.go:28] interesting pod/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 16 12:31:47.798818 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:47.798619    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5" podUID="eaa9624f-72d7-4a15-abcc-ba19f761388a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 16 12:31:48.798297 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:48.798257    2579 patch_prober.go:28] interesting pod/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 16 12:31:48.798800 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:48.798318    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5" podUID="eaa9624f-72d7-4a15-abcc-ba19f761388a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 16 12:31:48.891579 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:48.891537    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:49.799058 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:49.799018    2579 patch_prober.go:28] interesting pod/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 16 12:31:49.800052 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:49.799633    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5" podUID="eaa9624f-72d7-4a15-abcc-ba19f761388a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 16 12:31:50.802546 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:50.802511    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:50.803526 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:50.803503    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:31:53.964637 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:53.964605    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl"]
Sep 16 12:31:53.967760 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:53.967728    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl"
Sep 16 12:31:53.975381 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:53.975362    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-352217-serving-cert\""
Sep 16 12:31:53.999173 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:53.999144    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl"]
Sep 16 12:31:54.080958 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:54.080918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-maas-api-tls\") pod \"maas-api-e2e-api-b-352217-557456bcf8-qrfjl\" (UID: \"c511a6e4-dddc-4b5b-b7ce-11b31bb263ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl"
Sep 16 12:31:54.081121 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:54.081019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-openshift-service-ca\") pod \"maas-api-e2e-api-b-352217-557456bcf8-qrfjl\" (UID: \"c511a6e4-dddc-4b5b-b7ce-11b31bb263ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl"
Sep 16 12:31:54.081121 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:54.081087    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktv2p\" (UniqueName: \"kubernetes.io/projected/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-kube-api-access-ktv2p\") pod \"maas-api-e2e-api-b-352217-557456bcf8-qrfjl\" (UID: \"c511a6e4-dddc-4b5b-b7ce-11b31bb263ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl"
Sep 16 12:31:54.081211 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:54.081126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-metrics-certs\") pod \"maas-api-e2e-api-b-352217-557456bcf8-qrfjl\" (UID: \"c511a6e4-dddc-4b5b-b7ce-11b31bb263ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl"
Sep 16 12:31:54.182332 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:54.182291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-maas-api-tls\") pod \"maas-api-e2e-api-b-352217-557456bcf8-qrfjl\" (UID: \"c511a6e4-dddc-4b5b-b7ce-11b31bb263ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl"
Sep 16 12:31:54.182532 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:54.182355    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-openshift-service-ca\") pod \"maas-api-e2e-api-b-352217-557456bcf8-qrfjl\" (UID: \"c511a6e4-dddc-4b5b-b7ce-11b31bb263ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl"
Sep 16 12:31:54.182532 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:54.182394    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktv2p\" (UniqueName: \"kubernetes.io/projected/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-kube-api-access-ktv2p\") pod \"maas-api-e2e-api-b-352217-557456bcf8-qrfjl\" (UID: \"c511a6e4-dddc-4b5b-b7ce-11b31bb263ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl"
Sep 16 12:31:54.182532 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:54.182424    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-metrics-certs\") pod \"maas-api-e2e-api-b-352217-557456bcf8-qrfjl\" (UID: \"c511a6e4-dddc-4b5b-b7ce-11b31bb263ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl"
Sep 16 12:31:54.182532 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:31:54.182445    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-352217-serving-cert: secret "maas-api-e2e-api-b-352217-serving-cert" not found
Sep 16 12:31:54.182788 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:31:54.182545    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-maas-api-tls podName:c511a6e4-dddc-4b5b-b7ce-11b31bb263ae nodeName:}" failed. No retries permitted until 2026-09-16 12:31:54.682507974 +0000 UTC m=+1708.383579282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-maas-api-tls") pod "maas-api-e2e-api-b-352217-557456bcf8-qrfjl" (UID: "c511a6e4-dddc-4b5b-b7ce-11b31bb263ae") : secret "maas-api-e2e-api-b-352217-serving-cert" not found
Sep 16 12:31:54.183138 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:54.183116    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-openshift-service-ca\") pod \"maas-api-e2e-api-b-352217-557456bcf8-qrfjl\" (UID: \"c511a6e4-dddc-4b5b-b7ce-11b31bb263ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl"
Sep 16 12:31:54.185078 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:54.185058    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-metrics-certs\") pod \"maas-api-e2e-api-b-352217-557456bcf8-qrfjl\" (UID: \"c511a6e4-dddc-4b5b-b7ce-11b31bb263ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl"
Sep 16 12:31:54.201972 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:54.201943    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktv2p\" (UniqueName: \"kubernetes.io/projected/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-kube-api-access-ktv2p\") pod \"maas-api-e2e-api-b-352217-557456bcf8-qrfjl\" (UID: \"c511a6e4-dddc-4b5b-b7ce-11b31bb263ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl"
Sep 16 12:31:54.688741 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:54.688709    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-maas-api-tls\") pod \"maas-api-e2e-api-b-352217-557456bcf8-qrfjl\" (UID: \"c511a6e4-dddc-4b5b-b7ce-11b31bb263ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl"
Sep 16 12:31:54.691133 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:54.691104    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-maas-api-tls\") pod \"maas-api-e2e-api-b-352217-557456bcf8-qrfjl\" (UID: \"c511a6e4-dddc-4b5b-b7ce-11b31bb263ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl"
Sep 16 12:31:54.878751 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:54.878716    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl"
Sep 16 12:31:55.199808 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:31:55.199770    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc511a6e4_dddc_4b5b_b7ce_11b31bb263ae.slice/crio-379cf0bdcdc34cd70bf1417d7496b26aa957a0448d4f218bc1645f7f02212b67 WatchSource:0}: Error finding container 379cf0bdcdc34cd70bf1417d7496b26aa957a0448d4f218bc1645f7f02212b67: Status 404 returned error can't find the container with id 379cf0bdcdc34cd70bf1417d7496b26aa957a0448d4f218bc1645f7f02212b67
Sep 16 12:31:55.200133 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:55.200067    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl"]
Sep 16 12:31:55.245306 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:55.245276    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl" event={"ID":"c511a6e4-dddc-4b5b-b7ce-11b31bb263ae","Type":"ContainerStarted","Data":"379cf0bdcdc34cd70bf1417d7496b26aa957a0448d4f218bc1645f7f02212b67"}
Sep 16 12:31:56.253974 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:56.253934    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl" event={"ID":"c511a6e4-dddc-4b5b-b7ce-11b31bb263ae","Type":"ContainerStarted","Data":"1b2d341aebd2398059c552524e32029afff83b65cbe176518b2b45cb8cbaf286"}
Sep 16 12:31:56.254386 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:56.254032    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl"
Sep 16 12:31:56.304469 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:56.304419    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl" podStartSLOduration=2.986981095 podStartE2EDuration="3.304403501s" totalImagesPullingTime="317.422406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:31:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:31:56.302887614 +0000 UTC m=+1710.003958942" watchObservedRunningTime="2026-09-16 12:31:56.304403501 +0000 UTC m=+1710.005474827"
Sep 16 12:31:58.788575 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:58.788541    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc" podUID="b97355ae-45e5-4812-b0b1-3abc6d7a0c69" containerName="main" containerID="cri-o://6caf0bd52ba04f49cb8d6532eb529d31caa7d3c76d97985ddee1145fffad816b" gracePeriod=45
Sep 16 12:31:58.797520 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:58.797483    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc" podUID="b97355ae-45e5-4812-b0b1-3abc6d7a0c69" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": dial tcp 10.134.0.77:8000: connect: connection refused"
Sep 16 12:31:58.823449 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:58.823420    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh" podUID="9d2173d2-5881-4e76-8de6-548d486111db" containerName="main" containerID="cri-o://7d2dc0c293c23a239d30f2e70c075ed465330ebf01fce1872edc7d1570ccfd55" gracePeriod=45
Sep 16 12:31:59.111799 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.111775    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:31:59.115016 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.114992    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:31:59.144811 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.144782    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-tls-certs\") pod \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\" (UID: \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\") "
Sep 16 12:31:59.144965 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.144836    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d2173d2-5881-4e76-8de6-548d486111db-model-cache\") pod \"9d2173d2-5881-4e76-8de6-548d486111db\" (UID: \"9d2173d2-5881-4e76-8de6-548d486111db\") "
Sep 16 12:31:59.144965 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.144864    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9prbd\" (UniqueName: \"kubernetes.io/projected/9d2173d2-5881-4e76-8de6-548d486111db-kube-api-access-9prbd\") pod \"9d2173d2-5881-4e76-8de6-548d486111db\" (UID: \"9d2173d2-5881-4e76-8de6-548d486111db\") "
Sep 16 12:31:59.144965 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.144891    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d2173d2-5881-4e76-8de6-548d486111db-dshm\") pod \"9d2173d2-5881-4e76-8de6-548d486111db\" (UID: \"9d2173d2-5881-4e76-8de6-548d486111db\") "
Sep 16 12:31:59.144965 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.144917    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2173d2-5881-4e76-8de6-548d486111db-tmp-dir\") pod \"9d2173d2-5881-4e76-8de6-548d486111db\" (UID: \"9d2173d2-5881-4e76-8de6-548d486111db\") "
Sep 16 12:31:59.144965 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.144957    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-model-cache\") pod \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\" (UID: \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\") "
Sep 16 12:31:59.145290 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.145000    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-dshm\") pod \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\" (UID: \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\") "
Sep 16 12:31:59.145290 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.145036    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d2173d2-5881-4e76-8de6-548d486111db-home\") pod \"9d2173d2-5881-4e76-8de6-548d486111db\" (UID: \"9d2173d2-5881-4e76-8de6-548d486111db\") "
Sep 16 12:31:59.145290 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.145074    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tq7j\" (UniqueName: \"kubernetes.io/projected/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-kube-api-access-4tq7j\") pod \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\" (UID: \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\") "
Sep 16 12:31:59.145290 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.145113    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-home\") pod \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\" (UID: \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\") "
Sep 16 12:31:59.145290 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.145150    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d2173d2-5881-4e76-8de6-548d486111db-tls-certs\") pod \"9d2173d2-5881-4e76-8de6-548d486111db\" (UID: \"9d2173d2-5881-4e76-8de6-548d486111db\") "
Sep 16 12:31:59.145290 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.145182    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d2173d2-5881-4e76-8de6-548d486111db-model-cache" (OuterVolumeSpecName: "model-cache") pod "9d2173d2-5881-4e76-8de6-548d486111db" (UID: "9d2173d2-5881-4e76-8de6-548d486111db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:59.145290 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.145200    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-tmp-dir\") pod \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\" (UID: \"b97355ae-45e5-4812-b0b1-3abc6d7a0c69\") "
Sep 16 12:31:59.145647 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.145551    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d2173d2-5881-4e76-8de6-548d486111db-model-cache\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.145647 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.145193    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d2173d2-5881-4e76-8de6-548d486111db-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9d2173d2-5881-4e76-8de6-548d486111db" (UID: "9d2173d2-5881-4e76-8de6-548d486111db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:59.145647 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.145372    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d2173d2-5881-4e76-8de6-548d486111db-home" (OuterVolumeSpecName: "home") pod "9d2173d2-5881-4e76-8de6-548d486111db" (UID: "9d2173d2-5881-4e76-8de6-548d486111db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:59.145647 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.145407    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-model-cache" (OuterVolumeSpecName: "model-cache") pod "b97355ae-45e5-4812-b0b1-3abc6d7a0c69" (UID: "b97355ae-45e5-4812-b0b1-3abc6d7a0c69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:59.145647 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.145631    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-home" (OuterVolumeSpecName: "home") pod "b97355ae-45e5-4812-b0b1-3abc6d7a0c69" (UID: "b97355ae-45e5-4812-b0b1-3abc6d7a0c69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:59.145935 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.145834    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b97355ae-45e5-4812-b0b1-3abc6d7a0c69" (UID: "b97355ae-45e5-4812-b0b1-3abc6d7a0c69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:59.147828 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.147799    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d2173d2-5881-4e76-8de6-548d486111db-dshm" (OuterVolumeSpecName: "dshm") pod "9d2173d2-5881-4e76-8de6-548d486111db" (UID: "9d2173d2-5881-4e76-8de6-548d486111db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:59.147956 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.147804    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d2173d2-5881-4e76-8de6-548d486111db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d2173d2-5881-4e76-8de6-548d486111db" (UID: "9d2173d2-5881-4e76-8de6-548d486111db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:31:59.147956 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.147907    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d2173d2-5881-4e76-8de6-548d486111db-kube-api-access-9prbd" (OuterVolumeSpecName: "kube-api-access-9prbd") pod "9d2173d2-5881-4e76-8de6-548d486111db" (UID: "9d2173d2-5881-4e76-8de6-548d486111db"). InnerVolumeSpecName "kube-api-access-9prbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:31:59.148115 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.148088    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b97355ae-45e5-4812-b0b1-3abc6d7a0c69" (UID: "b97355ae-45e5-4812-b0b1-3abc6d7a0c69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:31:59.148211 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.148182    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-dshm" (OuterVolumeSpecName: "dshm") pod "b97355ae-45e5-4812-b0b1-3abc6d7a0c69" (UID: "b97355ae-45e5-4812-b0b1-3abc6d7a0c69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:31:59.148257 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.148213    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-kube-api-access-4tq7j" (OuterVolumeSpecName: "kube-api-access-4tq7j") pod "b97355ae-45e5-4812-b0b1-3abc6d7a0c69" (UID: "b97355ae-45e5-4812-b0b1-3abc6d7a0c69"). InnerVolumeSpecName "kube-api-access-4tq7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:31:59.246376 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.246343    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-tmp-dir\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.246376 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.246373    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-tls-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.246592 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.246389    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9prbd\" (UniqueName: \"kubernetes.io/projected/9d2173d2-5881-4e76-8de6-548d486111db-kube-api-access-9prbd\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.246592 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.246401    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d2173d2-5881-4e76-8de6-548d486111db-dshm\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.246592 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.246412    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2173d2-5881-4e76-8de6-548d486111db-tmp-dir\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.246592 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.246422    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-model-cache\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.246592 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.246433    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-dshm\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.246592 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.246443    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d2173d2-5881-4e76-8de6-548d486111db-home\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.246592 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.246455    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tq7j\" (UniqueName: \"kubernetes.io/projected/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-kube-api-access-4tq7j\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.246592 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.246465    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b97355ae-45e5-4812-b0b1-3abc6d7a0c69-home\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.246592 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.246475    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d2173d2-5881-4e76-8de6-548d486111db-tls-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:31:59.273569 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.273533    2579 generic.go:356] "Generic (PLEG): container finished" podID="9d2173d2-5881-4e76-8de6-548d486111db" containerID="7d2dc0c293c23a239d30f2e70c075ed465330ebf01fce1872edc7d1570ccfd55" exitCode=0
Sep 16 12:31:59.273776 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.273595    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"
Sep 16 12:31:59.273776 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.273619    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh" event={"ID":"9d2173d2-5881-4e76-8de6-548d486111db","Type":"ContainerDied","Data":"7d2dc0c293c23a239d30f2e70c075ed465330ebf01fce1872edc7d1570ccfd55"}
Sep 16 12:31:59.273776 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.273695    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh" event={"ID":"9d2173d2-5881-4e76-8de6-548d486111db","Type":"ContainerDied","Data":"addbf7a63fe2a07ba31a12ff473754a53f7772bf09fefb98e6a00938b454f1b7"}
Sep 16 12:31:59.273776 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.273722    2579 scope.go:117] "RemoveContainer" containerID="7d2dc0c293c23a239d30f2e70c075ed465330ebf01fce1872edc7d1570ccfd55"
Sep 16 12:31:59.275185 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.275166    2579 generic.go:356] "Generic (PLEG): container finished" podID="b97355ae-45e5-4812-b0b1-3abc6d7a0c69" containerID="6caf0bd52ba04f49cb8d6532eb529d31caa7d3c76d97985ddee1145fffad816b" exitCode=0
Sep 16 12:31:59.275278 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.275221    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"
Sep 16 12:31:59.275278 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.275252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc" event={"ID":"b97355ae-45e5-4812-b0b1-3abc6d7a0c69","Type":"ContainerDied","Data":"6caf0bd52ba04f49cb8d6532eb529d31caa7d3c76d97985ddee1145fffad816b"}
Sep 16 12:31:59.275392 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.275294    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc" event={"ID":"b97355ae-45e5-4812-b0b1-3abc6d7a0c69","Type":"ContainerDied","Data":"b8bd1f209731c09988d897ff0721eec9f6c840fe62c180960393c2bc8b0ff7cf"}
Sep 16 12:31:59.285221 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.285203    2579 scope.go:117] "RemoveContainer" containerID="7d2dc0c293c23a239d30f2e70c075ed465330ebf01fce1872edc7d1570ccfd55"
Sep 16 12:31:59.285540 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:31:59.285519    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d2dc0c293c23a239d30f2e70c075ed465330ebf01fce1872edc7d1570ccfd55\": container with ID starting with 7d2dc0c293c23a239d30f2e70c075ed465330ebf01fce1872edc7d1570ccfd55 not found: ID does not exist" containerID="7d2dc0c293c23a239d30f2e70c075ed465330ebf01fce1872edc7d1570ccfd55"
Sep 16 12:31:59.285632 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.285547    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d2dc0c293c23a239d30f2e70c075ed465330ebf01fce1872edc7d1570ccfd55"} err="failed to get container status \"7d2dc0c293c23a239d30f2e70c075ed465330ebf01fce1872edc7d1570ccfd55\": rpc error: code = NotFound desc = could not find container \"7d2dc0c293c23a239d30f2e70c075ed465330ebf01fce1872edc7d1570ccfd55\": container with ID starting with 7d2dc0c293c23a239d30f2e70c075ed465330ebf01fce1872edc7d1570ccfd55 not found: ID does not exist"
Sep 16 12:31:59.285632 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.285563    2579 scope.go:117] "RemoveContainer" containerID="6caf0bd52ba04f49cb8d6532eb529d31caa7d3c76d97985ddee1145fffad816b"
Sep 16 12:31:59.294038 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.294021    2579 scope.go:117] "RemoveContainer" containerID="6caf0bd52ba04f49cb8d6532eb529d31caa7d3c76d97985ddee1145fffad816b"
Sep 16 12:31:59.294264 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:31:59.294246    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6caf0bd52ba04f49cb8d6532eb529d31caa7d3c76d97985ddee1145fffad816b\": container with ID starting with 6caf0bd52ba04f49cb8d6532eb529d31caa7d3c76d97985ddee1145fffad816b not found: ID does not exist" containerID="6caf0bd52ba04f49cb8d6532eb529d31caa7d3c76d97985ddee1145fffad816b"
Sep 16 12:31:59.294315 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.294268    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6caf0bd52ba04f49cb8d6532eb529d31caa7d3c76d97985ddee1145fffad816b"} err="failed to get container status \"6caf0bd52ba04f49cb8d6532eb529d31caa7d3c76d97985ddee1145fffad816b\": rpc error: code = NotFound desc = could not find container \"6caf0bd52ba04f49cb8d6532eb529d31caa7d3c76d97985ddee1145fffad816b\": container with ID starting with 6caf0bd52ba04f49cb8d6532eb529d31caa7d3c76d97985ddee1145fffad816b not found: ID does not exist"
Sep 16 12:31:59.332403 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.332371    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"]
Sep 16 12:31:59.348031 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.347963    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/embedding-w0-739ee7-kserve-74b488956d-9rfsc"]
Sep 16 12:31:59.391635 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.391595    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"]
Sep 16 12:31:59.402727 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:31:59.402694    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-739ee7/premium-w0-739ee7-kserve-68d6ff5dcb-tt5qh"]
Sep 16 12:32:00.781718 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:00.781688    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d2173d2-5881-4e76-8de6-548d486111db" path="/var/lib/kubelet/pods/9d2173d2-5881-4e76-8de6-548d486111db/volumes"
Sep 16 12:32:00.782095 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:00.782085    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b97355ae-45e5-4812-b0b1-3abc6d7a0c69" path="/var/lib/kubelet/pods/b97355ae-45e5-4812-b0b1-3abc6d7a0c69/volumes"
Sep 16 12:32:01.795362 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:01.795329    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-739ee7-29826030-zpswx"]
Sep 16 12:32:01.804465 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:01.804439    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-739ee7-29826030-zpswx"]
Sep 16 12:32:02.265512 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:02.265481    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl"
Sep 16 12:32:02.782034 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:02.781998    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67e4a997-eacd-49df-8c6e-4298c01b640b" path="/var/lib/kubelet/pods/67e4a997-eacd-49df-8c6e-4298c01b640b/volumes"
Sep 16 12:32:11.873093 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:11.873051    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:32:14.916321 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:14.916282    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"]
Sep 16 12:32:14.916778 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:14.916495    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg" podUID="5ade63ee-3184-46df-ada2-55a290bcbefc" containerName="istio-proxy" containerID="cri-o://1077e9fbe6a5a07553e69cec535e3de9a5653d4b1be3440f71ea05b06a4ef89a" gracePeriod=30
Sep 16 12:32:15.244634 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:15.244548    2579 patch_prober.go:28] interesting pod/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 12:32:15.244634 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:15.244622    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg" podUID="5ade63ee-3184-46df-ada2-55a290bcbefc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:32:20.167358 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.167328    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:32:20.253699 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.253597    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-istio-data\") pod \"5ade63ee-3184-46df-ada2-55a290bcbefc\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") "
Sep 16 12:32:20.253699 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.253638    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ade63ee-3184-46df-ada2-55a290bcbefc-istio-token\") pod \"5ade63ee-3184-46df-ada2-55a290bcbefc\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") "
Sep 16 12:32:20.253930 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.253701    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-workload-socket\") pod \"5ade63ee-3184-46df-ada2-55a290bcbefc\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") "
Sep 16 12:32:20.253930 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.253723    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-workload-certs\") pod \"5ade63ee-3184-46df-ada2-55a290bcbefc\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") "
Sep 16 12:32:20.253930 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.253744    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szgdf\" (UniqueName: \"kubernetes.io/projected/5ade63ee-3184-46df-ada2-55a290bcbefc-kube-api-access-szgdf\") pod \"5ade63ee-3184-46df-ada2-55a290bcbefc\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") "
Sep 16 12:32:20.253930 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.253820    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ade63ee-3184-46df-ada2-55a290bcbefc-istio-podinfo\") pod \"5ade63ee-3184-46df-ada2-55a290bcbefc\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") "
Sep 16 12:32:20.253930 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.253890    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-istio-envoy\") pod \"5ade63ee-3184-46df-ada2-55a290bcbefc\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") "
Sep 16 12:32:20.253930 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.253912    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ade63ee-3184-46df-ada2-55a290bcbefc-istiod-ca-cert\") pod \"5ade63ee-3184-46df-ada2-55a290bcbefc\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") "
Sep 16 12:32:20.254262 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.253963    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-credential-socket\") pod \"5ade63ee-3184-46df-ada2-55a290bcbefc\" (UID: \"5ade63ee-3184-46df-ada2-55a290bcbefc\") "
Sep 16 12:32:20.254262 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.253975    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-istio-data" (OuterVolumeSpecName: "istio-data") pod "5ade63ee-3184-46df-ada2-55a290bcbefc" (UID: "5ade63ee-3184-46df-ada2-55a290bcbefc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:20.254262 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.253995    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5ade63ee-3184-46df-ada2-55a290bcbefc" (UID: "5ade63ee-3184-46df-ada2-55a290bcbefc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:20.254262 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.254016    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5ade63ee-3184-46df-ada2-55a290bcbefc" (UID: "5ade63ee-3184-46df-ada2-55a290bcbefc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:20.254480 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.254348    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-workload-socket\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:20.254480 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.254371    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-workload-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:20.254480 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.254386    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-istio-data\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:20.254480 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.254435    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5ade63ee-3184-46df-ada2-55a290bcbefc" (UID: "5ade63ee-3184-46df-ada2-55a290bcbefc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:20.254720 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.254509    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ade63ee-3184-46df-ada2-55a290bcbefc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5ade63ee-3184-46df-ada2-55a290bcbefc" (UID: "5ade63ee-3184-46df-ada2-55a290bcbefc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:32:20.256145 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.256117    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ade63ee-3184-46df-ada2-55a290bcbefc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5ade63ee-3184-46df-ada2-55a290bcbefc" (UID: "5ade63ee-3184-46df-ada2-55a290bcbefc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:32:20.256251 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.256187    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5ade63ee-3184-46df-ada2-55a290bcbefc" (UID: "5ade63ee-3184-46df-ada2-55a290bcbefc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:20.256251 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.256218    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ade63ee-3184-46df-ada2-55a290bcbefc-kube-api-access-szgdf" (OuterVolumeSpecName: "kube-api-access-szgdf") pod "5ade63ee-3184-46df-ada2-55a290bcbefc" (UID: "5ade63ee-3184-46df-ada2-55a290bcbefc"). InnerVolumeSpecName "kube-api-access-szgdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:32:20.256325 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.256256    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ade63ee-3184-46df-ada2-55a290bcbefc-istio-token" (OuterVolumeSpecName: "istio-token") pod "5ade63ee-3184-46df-ada2-55a290bcbefc" (UID: "5ade63ee-3184-46df-ada2-55a290bcbefc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:32:20.355970 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.355933    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ade63ee-3184-46df-ada2-55a290bcbefc-istio-podinfo\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:20.355970 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.355961    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-istio-envoy\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:20.355970 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.355971    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ade63ee-3184-46df-ada2-55a290bcbefc-istiod-ca-cert\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:20.355970 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.355979    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ade63ee-3184-46df-ada2-55a290bcbefc-credential-socket\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:20.356236 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.355988    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ade63ee-3184-46df-ada2-55a290bcbefc-istio-token\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:20.356236 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.355997    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szgdf\" (UniqueName: \"kubernetes.io/projected/5ade63ee-3184-46df-ada2-55a290bcbefc-kube-api-access-szgdf\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:20.408881 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.408842    2579 generic.go:356] "Generic (PLEG): container finished" podID="5ade63ee-3184-46df-ada2-55a290bcbefc" containerID="1077e9fbe6a5a07553e69cec535e3de9a5653d4b1be3440f71ea05b06a4ef89a" exitCode=0
Sep 16 12:32:20.409074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.408899    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"
Sep 16 12:32:20.409074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.408930    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg" event={"ID":"5ade63ee-3184-46df-ada2-55a290bcbefc","Type":"ContainerDied","Data":"1077e9fbe6a5a07553e69cec535e3de9a5653d4b1be3440f71ea05b06a4ef89a"}
Sep 16 12:32:20.409074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.409030    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg" event={"ID":"5ade63ee-3184-46df-ada2-55a290bcbefc","Type":"ContainerDied","Data":"d044b52a33f2fb4ce4ba3c3aad9389a1da5df265f7335a3a369c5b9de8e622b1"}
Sep 16 12:32:20.409074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.409062    2579 scope.go:117] "RemoveContainer" containerID="1077e9fbe6a5a07553e69cec535e3de9a5653d4b1be3440f71ea05b06a4ef89a"
Sep 16 12:32:20.419548 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.419530    2579 scope.go:117] "RemoveContainer" containerID="da5db95d12256e348e633bc22c6c62e9ece315dd0cb228c175d796fe6ac3f0b1"
Sep 16 12:32:20.428390 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.428374    2579 scope.go:117] "RemoveContainer" containerID="1077e9fbe6a5a07553e69cec535e3de9a5653d4b1be3440f71ea05b06a4ef89a"
Sep 16 12:32:20.428641 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:32:20.428621    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1077e9fbe6a5a07553e69cec535e3de9a5653d4b1be3440f71ea05b06a4ef89a\": container with ID starting with 1077e9fbe6a5a07553e69cec535e3de9a5653d4b1be3440f71ea05b06a4ef89a not found: ID does not exist" containerID="1077e9fbe6a5a07553e69cec535e3de9a5653d4b1be3440f71ea05b06a4ef89a"
Sep 16 12:32:20.428722 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.428650    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1077e9fbe6a5a07553e69cec535e3de9a5653d4b1be3440f71ea05b06a4ef89a"} err="failed to get container status \"1077e9fbe6a5a07553e69cec535e3de9a5653d4b1be3440f71ea05b06a4ef89a\": rpc error: code = NotFound desc = could not find container \"1077e9fbe6a5a07553e69cec535e3de9a5653d4b1be3440f71ea05b06a4ef89a\": container with ID starting with 1077e9fbe6a5a07553e69cec535e3de9a5653d4b1be3440f71ea05b06a4ef89a not found: ID does not exist"
Sep 16 12:32:20.428722 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.428685    2579 scope.go:117] "RemoveContainer" containerID="da5db95d12256e348e633bc22c6c62e9ece315dd0cb228c175d796fe6ac3f0b1"
Sep 16 12:32:20.428919 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:32:20.428905    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da5db95d12256e348e633bc22c6c62e9ece315dd0cb228c175d796fe6ac3f0b1\": container with ID starting with da5db95d12256e348e633bc22c6c62e9ece315dd0cb228c175d796fe6ac3f0b1 not found: ID does not exist" containerID="da5db95d12256e348e633bc22c6c62e9ece315dd0cb228c175d796fe6ac3f0b1"
Sep 16 12:32:20.428965 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.428923    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5db95d12256e348e633bc22c6c62e9ece315dd0cb228c175d796fe6ac3f0b1"} err="failed to get container status \"da5db95d12256e348e633bc22c6c62e9ece315dd0cb228c175d796fe6ac3f0b1\": rpc error: code = NotFound desc = could not find container \"da5db95d12256e348e633bc22c6c62e9ece315dd0cb228c175d796fe6ac3f0b1\": container with ID starting with da5db95d12256e348e633bc22c6c62e9ece315dd0cb228c175d796fe6ac3f0b1 not found: ID does not exist"
Sep 16 12:32:20.450226 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.450201    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"]
Sep 16 12:32:20.456867 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.456846    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-739ee7-openshift-default-6b558b95b8-tnstg"]
Sep 16 12:32:20.782270 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:20.782236    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ade63ee-3184-46df-ada2-55a290bcbefc" path="/var/lib/kubelet/pods/5ade63ee-3184-46df-ada2-55a290bcbefc/volumes"
Sep 16 12:32:31.466205 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:31.466167    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r"]
Sep 16 12:32:31.466630 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:31.466413    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r" podUID="b8cad60f-d2a0-4b30-a107-68f0bf623ae2" containerName="maas-api" containerID="cri-o://30f374b63531767015302049e81e0311f0e108a27ea3cfb095d90a84321f4754" gracePeriod=30
Sep 16 12:32:31.729720 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:31.729643    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r"
Sep 16 12:32:31.873762 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:31.873730    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8cad60f-d2a0-4b30-a107-68f0bf623ae2-openshift-service-ca\") pod \"b8cad60f-d2a0-4b30-a107-68f0bf623ae2\" (UID: \"b8cad60f-d2a0-4b30-a107-68f0bf623ae2\") "
Sep 16 12:32:31.873961 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:31.873795    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8cad60f-d2a0-4b30-a107-68f0bf623ae2-maas-api-tls\") pod \"b8cad60f-d2a0-4b30-a107-68f0bf623ae2\" (UID: \"b8cad60f-d2a0-4b30-a107-68f0bf623ae2\") "
Sep 16 12:32:31.873961 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:31.873880    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8cad60f-d2a0-4b30-a107-68f0bf623ae2-metrics-certs\") pod \"b8cad60f-d2a0-4b30-a107-68f0bf623ae2\" (UID: \"b8cad60f-d2a0-4b30-a107-68f0bf623ae2\") "
Sep 16 12:32:31.874129 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:31.874032    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vc7t7\" (UniqueName: \"kubernetes.io/projected/b8cad60f-d2a0-4b30-a107-68f0bf623ae2-kube-api-access-vc7t7\") pod \"b8cad60f-d2a0-4b30-a107-68f0bf623ae2\" (UID: \"b8cad60f-d2a0-4b30-a107-68f0bf623ae2\") "
Sep 16 12:32:31.874193 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:31.874124    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8cad60f-d2a0-4b30-a107-68f0bf623ae2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b8cad60f-d2a0-4b30-a107-68f0bf623ae2" (UID: "b8cad60f-d2a0-4b30-a107-68f0bf623ae2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:32:31.874427 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:31.874404    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8cad60f-d2a0-4b30-a107-68f0bf623ae2-openshift-service-ca\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:31.883236 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:31.882189    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8cad60f-d2a0-4b30-a107-68f0bf623ae2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b8cad60f-d2a0-4b30-a107-68f0bf623ae2" (UID: "b8cad60f-d2a0-4b30-a107-68f0bf623ae2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:32:31.909922 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:31.909882    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8cad60f-d2a0-4b30-a107-68f0bf623ae2-kube-api-access-vc7t7" (OuterVolumeSpecName: "kube-api-access-vc7t7") pod "b8cad60f-d2a0-4b30-a107-68f0bf623ae2" (UID: "b8cad60f-d2a0-4b30-a107-68f0bf623ae2"). InnerVolumeSpecName "kube-api-access-vc7t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:32:31.911090 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:31.911060    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8cad60f-d2a0-4b30-a107-68f0bf623ae2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b8cad60f-d2a0-4b30-a107-68f0bf623ae2" (UID: "b8cad60f-d2a0-4b30-a107-68f0bf623ae2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:32:31.975027 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:31.974992    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8cad60f-d2a0-4b30-a107-68f0bf623ae2-maas-api-tls\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:31.975027 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:31.975022    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8cad60f-d2a0-4b30-a107-68f0bf623ae2-metrics-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:31.975027 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:31.975033    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc7t7\" (UniqueName: \"kubernetes.io/projected/b8cad60f-d2a0-4b30-a107-68f0bf623ae2-kube-api-access-vc7t7\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:32.488523 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:32.488483    2579 generic.go:356] "Generic (PLEG): container finished" podID="b8cad60f-d2a0-4b30-a107-68f0bf623ae2" containerID="30f374b63531767015302049e81e0311f0e108a27ea3cfb095d90a84321f4754" exitCode=0
Sep 16 12:32:32.489024 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:32.488550    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r"
Sep 16 12:32:32.489024 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:32.488574    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r" event={"ID":"b8cad60f-d2a0-4b30-a107-68f0bf623ae2","Type":"ContainerDied","Data":"30f374b63531767015302049e81e0311f0e108a27ea3cfb095d90a84321f4754"}
Sep 16 12:32:32.489024 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:32.488721    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r" event={"ID":"b8cad60f-d2a0-4b30-a107-68f0bf623ae2","Type":"ContainerDied","Data":"dbc8ef9bf5284726828db6edbbeea3322662933a6a50706f431cc992e758c6f8"}
Sep 16 12:32:32.489024 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:32.488759    2579 scope.go:117] "RemoveContainer" containerID="30f374b63531767015302049e81e0311f0e108a27ea3cfb095d90a84321f4754"
Sep 16 12:32:32.499462 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:32.499440    2579 scope.go:117] "RemoveContainer" containerID="30f374b63531767015302049e81e0311f0e108a27ea3cfb095d90a84321f4754"
Sep 16 12:32:32.499797 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:32:32.499773    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f374b63531767015302049e81e0311f0e108a27ea3cfb095d90a84321f4754\": container with ID starting with 30f374b63531767015302049e81e0311f0e108a27ea3cfb095d90a84321f4754 not found: ID does not exist" containerID="30f374b63531767015302049e81e0311f0e108a27ea3cfb095d90a84321f4754"
Sep 16 12:32:32.499863 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:32.499804    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f374b63531767015302049e81e0311f0e108a27ea3cfb095d90a84321f4754"} err="failed to get container status \"30f374b63531767015302049e81e0311f0e108a27ea3cfb095d90a84321f4754\": rpc error: code = NotFound desc = could not find container \"30f374b63531767015302049e81e0311f0e108a27ea3cfb095d90a84321f4754\": container with ID starting with 30f374b63531767015302049e81e0311f0e108a27ea3cfb095d90a84321f4754 not found: ID does not exist"
Sep 16 12:32:32.525948 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:32.525911    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r"]
Sep 16 12:32:32.534630 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:32.534597    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8641cd-5b49f676d9-hh46r"]
Sep 16 12:32:32.780980 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:32.780882    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8cad60f-d2a0-4b30-a107-68f0bf623ae2" path="/var/lib/kubelet/pods/b8cad60f-d2a0-4b30-a107-68f0bf623ae2/volumes"
Sep 16 12:32:40.192839 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:40.192798    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"]
Sep 16 12:32:40.193321 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:40.193023    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5" podUID="eaa9624f-72d7-4a15-abcc-ba19f761388a" containerName="istio-proxy" containerID="cri-o://f5c4dc5189fb059dae6beb582b75b8f388bbb40aaa0bb6e4cf42b1723a03aefc" gracePeriod=30
Sep 16 12:32:45.445410 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.445384    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:32:45.522426 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.522341    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-workload-socket\") pod \"eaa9624f-72d7-4a15-abcc-ba19f761388a\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") "
Sep 16 12:32:45.522426 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.522405    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-istio-envoy\") pod \"eaa9624f-72d7-4a15-abcc-ba19f761388a\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") "
Sep 16 12:32:45.522426 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.522425    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-istio-data\") pod \"eaa9624f-72d7-4a15-abcc-ba19f761388a\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") "
Sep 16 12:32:45.522719 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.522518    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-workload-certs\") pod \"eaa9624f-72d7-4a15-abcc-ba19f761388a\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") "
Sep 16 12:32:45.522719 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.522566    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa9624f-72d7-4a15-abcc-ba19f761388a-istio-token\") pod \"eaa9624f-72d7-4a15-abcc-ba19f761388a\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") "
Sep 16 12:32:45.522719 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.522601    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-655sj\" (UniqueName: \"kubernetes.io/projected/eaa9624f-72d7-4a15-abcc-ba19f761388a-kube-api-access-655sj\") pod \"eaa9624f-72d7-4a15-abcc-ba19f761388a\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") "
Sep 16 12:32:45.522719 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.522625    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-credential-socket\") pod \"eaa9624f-72d7-4a15-abcc-ba19f761388a\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") "
Sep 16 12:32:45.522719 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.522683    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eaa9624f-72d7-4a15-abcc-ba19f761388a" (UID: "eaa9624f-72d7-4a15-abcc-ba19f761388a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:45.522719 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.522650    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa9624f-72d7-4a15-abcc-ba19f761388a-istio-podinfo\") pod \"eaa9624f-72d7-4a15-abcc-ba19f761388a\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") "
Sep 16 12:32:45.523047 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.522788    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa9624f-72d7-4a15-abcc-ba19f761388a-istiod-ca-cert\") pod \"eaa9624f-72d7-4a15-abcc-ba19f761388a\" (UID: \"eaa9624f-72d7-4a15-abcc-ba19f761388a\") "
Sep 16 12:32:45.523047 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.522818    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-istio-data" (OuterVolumeSpecName: "istio-data") pod "eaa9624f-72d7-4a15-abcc-ba19f761388a" (UID: "eaa9624f-72d7-4a15-abcc-ba19f761388a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:45.523047 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.522883    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eaa9624f-72d7-4a15-abcc-ba19f761388a" (UID: "eaa9624f-72d7-4a15-abcc-ba19f761388a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:45.523047 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.523006    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eaa9624f-72d7-4a15-abcc-ba19f761388a" (UID: "eaa9624f-72d7-4a15-abcc-ba19f761388a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:45.523234 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.523224    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-workload-socket\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:45.523290 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.523241    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-istio-data\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:45.523290 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.523255    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-workload-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:45.523290 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.523268    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-credential-socket\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:45.523398 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.523289    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaa9624f-72d7-4a15-abcc-ba19f761388a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eaa9624f-72d7-4a15-abcc-ba19f761388a" (UID: "eaa9624f-72d7-4a15-abcc-ba19f761388a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:32:45.524686 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.524638    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa9624f-72d7-4a15-abcc-ba19f761388a-istio-token" (OuterVolumeSpecName: "istio-token") pod "eaa9624f-72d7-4a15-abcc-ba19f761388a" (UID: "eaa9624f-72d7-4a15-abcc-ba19f761388a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:32:45.524855 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.524837    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa9624f-72d7-4a15-abcc-ba19f761388a-kube-api-access-655sj" (OuterVolumeSpecName: "kube-api-access-655sj") pod "eaa9624f-72d7-4a15-abcc-ba19f761388a" (UID: "eaa9624f-72d7-4a15-abcc-ba19f761388a"). InnerVolumeSpecName "kube-api-access-655sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:32:45.525084 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.525062    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eaa9624f-72d7-4a15-abcc-ba19f761388a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eaa9624f-72d7-4a15-abcc-ba19f761388a" (UID: "eaa9624f-72d7-4a15-abcc-ba19f761388a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:32:45.525084 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.525068    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eaa9624f-72d7-4a15-abcc-ba19f761388a" (UID: "eaa9624f-72d7-4a15-abcc-ba19f761388a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:32:45.570552 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.570512    2579 generic.go:356] "Generic (PLEG): container finished" podID="eaa9624f-72d7-4a15-abcc-ba19f761388a" containerID="f5c4dc5189fb059dae6beb582b75b8f388bbb40aaa0bb6e4cf42b1723a03aefc" exitCode=0
Sep 16 12:32:45.570736 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.570559    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"
Sep 16 12:32:45.570736 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.570552    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5" event={"ID":"eaa9624f-72d7-4a15-abcc-ba19f761388a","Type":"ContainerDied","Data":"f5c4dc5189fb059dae6beb582b75b8f388bbb40aaa0bb6e4cf42b1723a03aefc"}
Sep 16 12:32:45.570736 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.570704    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5" event={"ID":"eaa9624f-72d7-4a15-abcc-ba19f761388a","Type":"ContainerDied","Data":"874c6907188e0746ba4240c6326510506368a15971816ff837cf394bc8967e19"}
Sep 16 12:32:45.570898 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.570730    2579 scope.go:117] "RemoveContainer" containerID="f5c4dc5189fb059dae6beb582b75b8f388bbb40aaa0bb6e4cf42b1723a03aefc"
Sep 16 12:32:45.581132 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.581113    2579 scope.go:117] "RemoveContainer" containerID="7095f064a0dcda6d97873121788f0dcbc9ac21b08f71f09772e25cf45e21aaa9"
Sep 16 12:32:45.589469 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.589452    2579 scope.go:117] "RemoveContainer" containerID="f5c4dc5189fb059dae6beb582b75b8f388bbb40aaa0bb6e4cf42b1723a03aefc"
Sep 16 12:32:45.589781 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:32:45.589762    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c4dc5189fb059dae6beb582b75b8f388bbb40aaa0bb6e4cf42b1723a03aefc\": container with ID starting with f5c4dc5189fb059dae6beb582b75b8f388bbb40aaa0bb6e4cf42b1723a03aefc not found: ID does not exist" containerID="f5c4dc5189fb059dae6beb582b75b8f388bbb40aaa0bb6e4cf42b1723a03aefc"
Sep 16 12:32:45.589820 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.589789    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c4dc5189fb059dae6beb582b75b8f388bbb40aaa0bb6e4cf42b1723a03aefc"} err="failed to get container status \"f5c4dc5189fb059dae6beb582b75b8f388bbb40aaa0bb6e4cf42b1723a03aefc\": rpc error: code = NotFound desc = could not find container \"f5c4dc5189fb059dae6beb582b75b8f388bbb40aaa0bb6e4cf42b1723a03aefc\": container with ID starting with f5c4dc5189fb059dae6beb582b75b8f388bbb40aaa0bb6e4cf42b1723a03aefc not found: ID does not exist"
Sep 16 12:32:45.589820 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.589808    2579 scope.go:117] "RemoveContainer" containerID="7095f064a0dcda6d97873121788f0dcbc9ac21b08f71f09772e25cf45e21aaa9"
Sep 16 12:32:45.590041 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:32:45.590024    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7095f064a0dcda6d97873121788f0dcbc9ac21b08f71f09772e25cf45e21aaa9\": container with ID starting with 7095f064a0dcda6d97873121788f0dcbc9ac21b08f71f09772e25cf45e21aaa9 not found: ID does not exist" containerID="7095f064a0dcda6d97873121788f0dcbc9ac21b08f71f09772e25cf45e21aaa9"
Sep 16 12:32:45.590080 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.590048    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7095f064a0dcda6d97873121788f0dcbc9ac21b08f71f09772e25cf45e21aaa9"} err="failed to get container status \"7095f064a0dcda6d97873121788f0dcbc9ac21b08f71f09772e25cf45e21aaa9\": rpc error: code = NotFound desc = could not find container \"7095f064a0dcda6d97873121788f0dcbc9ac21b08f71f09772e25cf45e21aaa9\": container with ID starting with 7095f064a0dcda6d97873121788f0dcbc9ac21b08f71f09772e25cf45e21aaa9 not found: ID does not exist"
Sep 16 12:32:45.610885 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.610860    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"]
Sep 16 12:32:45.613728 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.613705    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-8641cd-openshift-default-59659946bd-8bkc5"]
Sep 16 12:32:45.623975 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.623949    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa9624f-72d7-4a15-abcc-ba19f761388a-istio-token\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:45.623975 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.623972    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-655sj\" (UniqueName: \"kubernetes.io/projected/eaa9624f-72d7-4a15-abcc-ba19f761388a-kube-api-access-655sj\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:45.624109 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.623983    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa9624f-72d7-4a15-abcc-ba19f761388a-istio-podinfo\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:45.624109 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.623993    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa9624f-72d7-4a15-abcc-ba19f761388a-istiod-ca-cert\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:45.624109 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:45.624000    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa9624f-72d7-4a15-abcc-ba19f761388a-istio-envoy\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:46.781396 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:46.781366    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaa9624f-72d7-4a15-abcc-ba19f761388a" path="/var/lib/kubelet/pods/eaa9624f-72d7-4a15-abcc-ba19f761388a/volumes"
Sep 16 12:32:48.563064 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:48.563024    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:32:55.885161 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:55.885119    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl"]
Sep 16 12:32:55.885560 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:55.885333    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl" podUID="c511a6e4-dddc-4b5b-b7ce-11b31bb263ae" containerName="maas-api" containerID="cri-o://1b2d341aebd2398059c552524e32029afff83b65cbe176518b2b45cb8cbaf286" gracePeriod=30
Sep 16 12:32:56.157422 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:56.157393    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl"
Sep 16 12:32:56.226002 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:56.225972    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-maas-api-tls\") pod \"c511a6e4-dddc-4b5b-b7ce-11b31bb263ae\" (UID: \"c511a6e4-dddc-4b5b-b7ce-11b31bb263ae\") "
Sep 16 12:32:56.226168 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:56.226071    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-metrics-certs\") pod \"c511a6e4-dddc-4b5b-b7ce-11b31bb263ae\" (UID: \"c511a6e4-dddc-4b5b-b7ce-11b31bb263ae\") "
Sep 16 12:32:56.226168 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:56.226099    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-openshift-service-ca\") pod \"c511a6e4-dddc-4b5b-b7ce-11b31bb263ae\" (UID: \"c511a6e4-dddc-4b5b-b7ce-11b31bb263ae\") "
Sep 16 12:32:56.226259 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:56.226168    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktv2p\" (UniqueName: \"kubernetes.io/projected/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-kube-api-access-ktv2p\") pod \"c511a6e4-dddc-4b5b-b7ce-11b31bb263ae\" (UID: \"c511a6e4-dddc-4b5b-b7ce-11b31bb263ae\") "
Sep 16 12:32:56.226589 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:56.226556    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c511a6e4-dddc-4b5b-b7ce-11b31bb263ae" (UID: "c511a6e4-dddc-4b5b-b7ce-11b31bb263ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:32:56.228048 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:56.228019    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c511a6e4-dddc-4b5b-b7ce-11b31bb263ae" (UID: "c511a6e4-dddc-4b5b-b7ce-11b31bb263ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:32:56.228186 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:56.228163    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c511a6e4-dddc-4b5b-b7ce-11b31bb263ae" (UID: "c511a6e4-dddc-4b5b-b7ce-11b31bb263ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:32:56.228292 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:56.228264    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-kube-api-access-ktv2p" (OuterVolumeSpecName: "kube-api-access-ktv2p") pod "c511a6e4-dddc-4b5b-b7ce-11b31bb263ae" (UID: "c511a6e4-dddc-4b5b-b7ce-11b31bb263ae"). InnerVolumeSpecName "kube-api-access-ktv2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:32:56.327480 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:56.327444    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-maas-api-tls\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:56.327480 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:56.327471    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-metrics-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:56.327480 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:56.327480    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-openshift-service-ca\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:56.327480 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:56.327490    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktv2p\" (UniqueName: \"kubernetes.io/projected/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae-kube-api-access-ktv2p\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:32:56.638277 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:56.638238    2579 generic.go:356] "Generic (PLEG): container finished" podID="c511a6e4-dddc-4b5b-b7ce-11b31bb263ae" containerID="1b2d341aebd2398059c552524e32029afff83b65cbe176518b2b45cb8cbaf286" exitCode=0
Sep 16 12:32:56.638446 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:56.638288    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl"
Sep 16 12:32:56.638446 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:56.638326    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl" event={"ID":"c511a6e4-dddc-4b5b-b7ce-11b31bb263ae","Type":"ContainerDied","Data":"1b2d341aebd2398059c552524e32029afff83b65cbe176518b2b45cb8cbaf286"}
Sep 16 12:32:56.638446 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:56.638384    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl" event={"ID":"c511a6e4-dddc-4b5b-b7ce-11b31bb263ae","Type":"ContainerDied","Data":"379cf0bdcdc34cd70bf1417d7496b26aa957a0448d4f218bc1645f7f02212b67"}
Sep 16 12:32:56.638446 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:56.638408    2579 scope.go:117] "RemoveContainer" containerID="1b2d341aebd2398059c552524e32029afff83b65cbe176518b2b45cb8cbaf286"
Sep 16 12:32:56.648531 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:56.648512    2579 scope.go:117] "RemoveContainer" containerID="1b2d341aebd2398059c552524e32029afff83b65cbe176518b2b45cb8cbaf286"
Sep 16 12:32:56.648843 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:32:56.648821    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b2d341aebd2398059c552524e32029afff83b65cbe176518b2b45cb8cbaf286\": container with ID starting with 1b2d341aebd2398059c552524e32029afff83b65cbe176518b2b45cb8cbaf286 not found: ID does not exist" containerID="1b2d341aebd2398059c552524e32029afff83b65cbe176518b2b45cb8cbaf286"
Sep 16 12:32:56.648901 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:56.648852    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b2d341aebd2398059c552524e32029afff83b65cbe176518b2b45cb8cbaf286"} err="failed to get container status \"1b2d341aebd2398059c552524e32029afff83b65cbe176518b2b45cb8cbaf286\": rpc error: code = NotFound desc = could not find container \"1b2d341aebd2398059c552524e32029afff83b65cbe176518b2b45cb8cbaf286\": container with ID starting with 1b2d341aebd2398059c552524e32029afff83b65cbe176518b2b45cb8cbaf286 not found: ID does not exist"
Sep 16 12:32:56.684984 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:56.684950    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl"]
Sep 16 12:32:56.694412 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:56.694381    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-352217-557456bcf8-qrfjl"]
Sep 16 12:32:56.782990 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:32:56.782950    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c511a6e4-dddc-4b5b-b7ce-11b31bb263ae" path="/var/lib/kubelet/pods/c511a6e4-dddc-4b5b-b7ce-11b31bb263ae/volumes"
Sep 16 12:33:14.242906 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:14.242870    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:33:18.044209 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:18.044176    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl"]
Sep 16 12:33:18.768376 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:18.768338    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl" podUID="5c7be173-8ad3-42ce-ab28-c5f73f349438" containerName="maas-api" containerID="cri-o://bcc271a8180652366dadba181e14f89452a7a7405f296db8babaece027ac53bf" gracePeriod=30
Sep 16 12:33:18.804080 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:18.804035    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl" podUID="5c7be173-8ad3-42ce-ab28-c5f73f349438" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.63:8443/health\": dial tcp 10.134.0.63:8443: connect: connection refused"
Sep 16 12:33:19.020637 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:19.020570    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl"
Sep 16 12:33:19.055423 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:19.055396    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c7be173-8ad3-42ce-ab28-c5f73f349438-openshift-service-ca\") pod \"5c7be173-8ad3-42ce-ab28-c5f73f349438\" (UID: \"5c7be173-8ad3-42ce-ab28-c5f73f349438\") "
Sep 16 12:33:19.055870 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:19.055461    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7be173-8ad3-42ce-ab28-c5f73f349438-metrics-certs\") pod \"5c7be173-8ad3-42ce-ab28-c5f73f349438\" (UID: \"5c7be173-8ad3-42ce-ab28-c5f73f349438\") "
Sep 16 12:33:19.055870 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:19.055514    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c7be173-8ad3-42ce-ab28-c5f73f349438-maas-api-tls\") pod \"5c7be173-8ad3-42ce-ab28-c5f73f349438\" (UID: \"5c7be173-8ad3-42ce-ab28-c5f73f349438\") "
Sep 16 12:33:19.055870 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:19.055539    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4z829\" (UniqueName: \"kubernetes.io/projected/5c7be173-8ad3-42ce-ab28-c5f73f349438-kube-api-access-4z829\") pod \"5c7be173-8ad3-42ce-ab28-c5f73f349438\" (UID: \"5c7be173-8ad3-42ce-ab28-c5f73f349438\") "
Sep 16 12:33:19.055870 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:19.055738    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c7be173-8ad3-42ce-ab28-c5f73f349438-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5c7be173-8ad3-42ce-ab28-c5f73f349438" (UID: "5c7be173-8ad3-42ce-ab28-c5f73f349438"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:33:19.057633 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:19.057607    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c7be173-8ad3-42ce-ab28-c5f73f349438-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5c7be173-8ad3-42ce-ab28-c5f73f349438" (UID: "5c7be173-8ad3-42ce-ab28-c5f73f349438"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:33:19.057749 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:19.057713    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c7be173-8ad3-42ce-ab28-c5f73f349438-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5c7be173-8ad3-42ce-ab28-c5f73f349438" (UID: "5c7be173-8ad3-42ce-ab28-c5f73f349438"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:33:19.057801 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:19.057747    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7be173-8ad3-42ce-ab28-c5f73f349438-kube-api-access-4z829" (OuterVolumeSpecName: "kube-api-access-4z829") pod "5c7be173-8ad3-42ce-ab28-c5f73f349438" (UID: "5c7be173-8ad3-42ce-ab28-c5f73f349438"). InnerVolumeSpecName "kube-api-access-4z829". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:33:19.156794 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:19.156757    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c7be173-8ad3-42ce-ab28-c5f73f349438-openshift-service-ca\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:33:19.156794 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:19.156787    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7be173-8ad3-42ce-ab28-c5f73f349438-metrics-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:33:19.156794 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:19.156799    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c7be173-8ad3-42ce-ab28-c5f73f349438-maas-api-tls\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:33:19.157026 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:19.156810    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z829\" (UniqueName: \"kubernetes.io/projected/5c7be173-8ad3-42ce-ab28-c5f73f349438-kube-api-access-4z829\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:33:19.775617 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:19.775579    2579 generic.go:356] "Generic (PLEG): container finished" podID="5c7be173-8ad3-42ce-ab28-c5f73f349438" containerID="bcc271a8180652366dadba181e14f89452a7a7405f296db8babaece027ac53bf" exitCode=0
Sep 16 12:33:19.775617 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:19.775620    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl"
Sep 16 12:33:19.775895 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:19.775645    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl" event={"ID":"5c7be173-8ad3-42ce-ab28-c5f73f349438","Type":"ContainerDied","Data":"bcc271a8180652366dadba181e14f89452a7a7405f296db8babaece027ac53bf"}
Sep 16 12:33:19.775895 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:19.775710    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl" event={"ID":"5c7be173-8ad3-42ce-ab28-c5f73f349438","Type":"ContainerDied","Data":"7fa4301c2954625b8b309fba2ae226358a3e5028b581aaa05e44e850d6243193"}
Sep 16 12:33:19.775895 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:19.775737    2579 scope.go:117] "RemoveContainer" containerID="bcc271a8180652366dadba181e14f89452a7a7405f296db8babaece027ac53bf"
Sep 16 12:33:19.786089 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:19.786070    2579 scope.go:117] "RemoveContainer" containerID="bcc271a8180652366dadba181e14f89452a7a7405f296db8babaece027ac53bf"
Sep 16 12:33:19.786351 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:33:19.786330    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcc271a8180652366dadba181e14f89452a7a7405f296db8babaece027ac53bf\": container with ID starting with bcc271a8180652366dadba181e14f89452a7a7405f296db8babaece027ac53bf not found: ID does not exist" containerID="bcc271a8180652366dadba181e14f89452a7a7405f296db8babaece027ac53bf"
Sep 16 12:33:19.786396 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:19.786360    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcc271a8180652366dadba181e14f89452a7a7405f296db8babaece027ac53bf"} err="failed to get container status \"bcc271a8180652366dadba181e14f89452a7a7405f296db8babaece027ac53bf\": rpc error: code = NotFound desc = could not find container \"bcc271a8180652366dadba181e14f89452a7a7405f296db8babaece027ac53bf\": container with ID starting with bcc271a8180652366dadba181e14f89452a7a7405f296db8babaece027ac53bf not found: ID does not exist"
Sep 16 12:33:19.818459 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:19.818430    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl"]
Sep 16 12:33:19.834880 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:19.834742    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-cadd00-59c4f969d-lwgbl"]
Sep 16 12:33:20.781845 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:20.781802    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c7be173-8ad3-42ce-ab28-c5f73f349438" path="/var/lib/kubelet/pods/5c7be173-8ad3-42ce-ab28-c5f73f349438/volumes"
Sep 16 12:33:40.835237 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:40.835195    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:33:45.198091 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:45.198054    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4"]
Sep 16 12:33:45.198514 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:45.198291    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4" podUID="787efc7d-197f-4c85-9266-7e38ffaf83db" containerName="maas-api" containerID="cri-o://77f922b4a277740ddd07a84449e075ebd0a2b2918dbc3082149692dd94b1e1ce" gracePeriod=30
Sep 16 12:33:45.457467 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:45.457401    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4"
Sep 16 12:33:45.605697 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:45.605649    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/787efc7d-197f-4c85-9266-7e38ffaf83db-maas-api-tls\") pod \"787efc7d-197f-4c85-9266-7e38ffaf83db\" (UID: \"787efc7d-197f-4c85-9266-7e38ffaf83db\") "
Sep 16 12:33:45.605897 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:45.605722    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/787efc7d-197f-4c85-9266-7e38ffaf83db-openshift-service-ca\") pod \"787efc7d-197f-4c85-9266-7e38ffaf83db\" (UID: \"787efc7d-197f-4c85-9266-7e38ffaf83db\") "
Sep 16 12:33:45.605897 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:45.605802    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwn5q\" (UniqueName: \"kubernetes.io/projected/787efc7d-197f-4c85-9266-7e38ffaf83db-kube-api-access-mwn5q\") pod \"787efc7d-197f-4c85-9266-7e38ffaf83db\" (UID: \"787efc7d-197f-4c85-9266-7e38ffaf83db\") "
Sep 16 12:33:45.605897 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:45.605837    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/787efc7d-197f-4c85-9266-7e38ffaf83db-metrics-certs\") pod \"787efc7d-197f-4c85-9266-7e38ffaf83db\" (UID: \"787efc7d-197f-4c85-9266-7e38ffaf83db\") "
Sep 16 12:33:45.606151 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:45.606119    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/787efc7d-197f-4c85-9266-7e38ffaf83db-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "787efc7d-197f-4c85-9266-7e38ffaf83db" (UID: "787efc7d-197f-4c85-9266-7e38ffaf83db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:33:45.607947 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:45.607921    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/787efc7d-197f-4c85-9266-7e38ffaf83db-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "787efc7d-197f-4c85-9266-7e38ffaf83db" (UID: "787efc7d-197f-4c85-9266-7e38ffaf83db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:33:45.608047 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:45.607990    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/787efc7d-197f-4c85-9266-7e38ffaf83db-kube-api-access-mwn5q" (OuterVolumeSpecName: "kube-api-access-mwn5q") pod "787efc7d-197f-4c85-9266-7e38ffaf83db" (UID: "787efc7d-197f-4c85-9266-7e38ffaf83db"). InnerVolumeSpecName "kube-api-access-mwn5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:33:45.608047 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:45.607999    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/787efc7d-197f-4c85-9266-7e38ffaf83db-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "787efc7d-197f-4c85-9266-7e38ffaf83db" (UID: "787efc7d-197f-4c85-9266-7e38ffaf83db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:33:45.706906 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:45.706868    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwn5q\" (UniqueName: \"kubernetes.io/projected/787efc7d-197f-4c85-9266-7e38ffaf83db-kube-api-access-mwn5q\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:33:45.706906 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:45.706899    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/787efc7d-197f-4c85-9266-7e38ffaf83db-metrics-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:33:45.706906 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:45.706908    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/787efc7d-197f-4c85-9266-7e38ffaf83db-maas-api-tls\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:33:45.707143 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:45.706920    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/787efc7d-197f-4c85-9266-7e38ffaf83db-openshift-service-ca\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:33:45.926844 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:45.926799    2579 generic.go:356] "Generic (PLEG): container finished" podID="787efc7d-197f-4c85-9266-7e38ffaf83db" containerID="77f922b4a277740ddd07a84449e075ebd0a2b2918dbc3082149692dd94b1e1ce" exitCode=0
Sep 16 12:33:45.926844 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:45.926848    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4"
Sep 16 12:33:45.927061 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:45.926866    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4" event={"ID":"787efc7d-197f-4c85-9266-7e38ffaf83db","Type":"ContainerDied","Data":"77f922b4a277740ddd07a84449e075ebd0a2b2918dbc3082149692dd94b1e1ce"}
Sep 16 12:33:45.927061 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:45.926921    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4" event={"ID":"787efc7d-197f-4c85-9266-7e38ffaf83db","Type":"ContainerDied","Data":"10a24ffceb5b4a4026395237446f8de188ad09a279a1a4a67185fca44bfe6403"}
Sep 16 12:33:45.927061 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:45.926946    2579 scope.go:117] "RemoveContainer" containerID="77f922b4a277740ddd07a84449e075ebd0a2b2918dbc3082149692dd94b1e1ce"
Sep 16 12:33:45.937391 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:45.937367    2579 scope.go:117] "RemoveContainer" containerID="77f922b4a277740ddd07a84449e075ebd0a2b2918dbc3082149692dd94b1e1ce"
Sep 16 12:33:45.937683 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:33:45.937637    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77f922b4a277740ddd07a84449e075ebd0a2b2918dbc3082149692dd94b1e1ce\": container with ID starting with 77f922b4a277740ddd07a84449e075ebd0a2b2918dbc3082149692dd94b1e1ce not found: ID does not exist" containerID="77f922b4a277740ddd07a84449e075ebd0a2b2918dbc3082149692dd94b1e1ce"
Sep 16 12:33:45.937757 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:45.937690    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77f922b4a277740ddd07a84449e075ebd0a2b2918dbc3082149692dd94b1e1ce"} err="failed to get container status \"77f922b4a277740ddd07a84449e075ebd0a2b2918dbc3082149692dd94b1e1ce\": rpc error: code = NotFound desc = could not find container \"77f922b4a277740ddd07a84449e075ebd0a2b2918dbc3082149692dd94b1e1ce\": container with ID starting with 77f922b4a277740ddd07a84449e075ebd0a2b2918dbc3082149692dd94b1e1ce not found: ID does not exist"
Sep 16 12:33:45.961717 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:45.961687    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4"]
Sep 16 12:33:45.978278 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:45.978253    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-51ae50-68d6f7fd7d-h8mr4"]
Sep 16 12:33:46.782129 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:46.782088    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="787efc7d-197f-4c85-9266-7e38ffaf83db" path="/var/lib/kubelet/pods/787efc7d-197f-4c85-9266-7e38ffaf83db/volumes"
Sep 16 12:33:54.956310 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:54.956274    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"]
Sep 16 12:33:54.956854 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:54.956551    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7" podUID="33b4eb96-7651-48ab-9256-3614eeb10f6c" containerName="istio-proxy" containerID="cri-o://3acf5ac04b96cfe3d2a2fa9a328eace9aa2d8b682408e3c0833552b5ac1e6667" gracePeriod=30
Sep 16 12:33:56.523099 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:56.523065    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 12:33:56.523565 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:56.523141    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7" podUID="33b4eb96-7651-48ab-9256-3614eeb10f6c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:33:58.390788 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:33:58.390752    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:34:00.215102 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.215076    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:34:00.350601 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.350568    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b4eb96-7651-48ab-9256-3614eeb10f6c-istio-token\") pod \"33b4eb96-7651-48ab-9256-3614eeb10f6c\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") "
Sep 16 12:34:00.350601 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.350611    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b4eb96-7651-48ab-9256-3614eeb10f6c-istio-podinfo\") pod \"33b4eb96-7651-48ab-9256-3614eeb10f6c\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") "
Sep 16 12:34:00.350891 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.350648    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7ggl\" (UniqueName: \"kubernetes.io/projected/33b4eb96-7651-48ab-9256-3614eeb10f6c-kube-api-access-s7ggl\") pod \"33b4eb96-7651-48ab-9256-3614eeb10f6c\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") "
Sep 16 12:34:00.350891 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.350703    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-workload-socket\") pod \"33b4eb96-7651-48ab-9256-3614eeb10f6c\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") "
Sep 16 12:34:00.350891 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.350748    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-credential-socket\") pod \"33b4eb96-7651-48ab-9256-3614eeb10f6c\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") "
Sep 16 12:34:00.350891 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.350770    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-istio-envoy\") pod \"33b4eb96-7651-48ab-9256-3614eeb10f6c\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") "
Sep 16 12:34:00.350891 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.350799    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-istio-data\") pod \"33b4eb96-7651-48ab-9256-3614eeb10f6c\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") "
Sep 16 12:34:00.350891 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.350887    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-workload-certs\") pod \"33b4eb96-7651-48ab-9256-3614eeb10f6c\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") "
Sep 16 12:34:00.351209 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.350916    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b4eb96-7651-48ab-9256-3614eeb10f6c-istiod-ca-cert\") pod \"33b4eb96-7651-48ab-9256-3614eeb10f6c\" (UID: \"33b4eb96-7651-48ab-9256-3614eeb10f6c\") "
Sep 16 12:34:00.351209 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.351102    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "33b4eb96-7651-48ab-9256-3614eeb10f6c" (UID: "33b4eb96-7651-48ab-9256-3614eeb10f6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:34:00.351317 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.351245    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "33b4eb96-7651-48ab-9256-3614eeb10f6c" (UID: "33b4eb96-7651-48ab-9256-3614eeb10f6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:34:00.351317 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.351293    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-credential-socket\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:34:00.351317 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.351263    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-istio-data" (OuterVolumeSpecName: "istio-data") pod "33b4eb96-7651-48ab-9256-3614eeb10f6c" (UID: "33b4eb96-7651-48ab-9256-3614eeb10f6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:34:00.351481 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.351379    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "33b4eb96-7651-48ab-9256-3614eeb10f6c" (UID: "33b4eb96-7651-48ab-9256-3614eeb10f6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:34:00.351481 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.351421    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b4eb96-7651-48ab-9256-3614eeb10f6c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "33b4eb96-7651-48ab-9256-3614eeb10f6c" (UID: "33b4eb96-7651-48ab-9256-3614eeb10f6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:34:00.352978 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.352945    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "33b4eb96-7651-48ab-9256-3614eeb10f6c" (UID: "33b4eb96-7651-48ab-9256-3614eeb10f6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:34:00.353104 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.352998    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b4eb96-7651-48ab-9256-3614eeb10f6c-kube-api-access-s7ggl" (OuterVolumeSpecName: "kube-api-access-s7ggl") pod "33b4eb96-7651-48ab-9256-3614eeb10f6c" (UID: "33b4eb96-7651-48ab-9256-3614eeb10f6c"). InnerVolumeSpecName "kube-api-access-s7ggl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:34:00.353104 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.352998    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b4eb96-7651-48ab-9256-3614eeb10f6c-istio-token" (OuterVolumeSpecName: "istio-token") pod "33b4eb96-7651-48ab-9256-3614eeb10f6c" (UID: "33b4eb96-7651-48ab-9256-3614eeb10f6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:34:00.353104 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.353045    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33b4eb96-7651-48ab-9256-3614eeb10f6c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "33b4eb96-7651-48ab-9256-3614eeb10f6c" (UID: "33b4eb96-7651-48ab-9256-3614eeb10f6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:34:00.452374 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.452330    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-workload-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:34:00.452374 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.452368    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b4eb96-7651-48ab-9256-3614eeb10f6c-istiod-ca-cert\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:34:00.452374 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.452377    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b4eb96-7651-48ab-9256-3614eeb10f6c-istio-token\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:34:00.452374 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.452385    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b4eb96-7651-48ab-9256-3614eeb10f6c-istio-podinfo\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:34:00.452684 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.452393    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7ggl\" (UniqueName: \"kubernetes.io/projected/33b4eb96-7651-48ab-9256-3614eeb10f6c-kube-api-access-s7ggl\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:34:00.452684 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.452405    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-workload-socket\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:34:00.452684 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.452420    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-istio-envoy\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:34:00.452684 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:00.452430    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b4eb96-7651-48ab-9256-3614eeb10f6c-istio-data\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:34:01.017231 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:01.017193    2579 generic.go:356] "Generic (PLEG): container finished" podID="33b4eb96-7651-48ab-9256-3614eeb10f6c" containerID="3acf5ac04b96cfe3d2a2fa9a328eace9aa2d8b682408e3c0833552b5ac1e6667" exitCode=0
Sep 16 12:34:01.017393 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:01.017245    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"
Sep 16 12:34:01.017393 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:01.017278    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7" event={"ID":"33b4eb96-7651-48ab-9256-3614eeb10f6c","Type":"ContainerDied","Data":"3acf5ac04b96cfe3d2a2fa9a328eace9aa2d8b682408e3c0833552b5ac1e6667"}
Sep 16 12:34:01.017393 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:01.017339    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7" event={"ID":"33b4eb96-7651-48ab-9256-3614eeb10f6c","Type":"ContainerDied","Data":"0c03794086ec082fc4d7e313f342e4c82437e865a425d2297ce99992529fc4da"}
Sep 16 12:34:01.017393 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:01.017368    2579 scope.go:117] "RemoveContainer" containerID="3acf5ac04b96cfe3d2a2fa9a328eace9aa2d8b682408e3c0833552b5ac1e6667"
Sep 16 12:34:01.026820 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:01.026800    2579 scope.go:117] "RemoveContainer" containerID="c7f6d20ddf4275b3aa4a9a725da93409fbe273ae93cd796439740aea9be792ed"
Sep 16 12:34:01.035881 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:01.035860    2579 scope.go:117] "RemoveContainer" containerID="3acf5ac04b96cfe3d2a2fa9a328eace9aa2d8b682408e3c0833552b5ac1e6667"
Sep 16 12:34:01.036198 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:34:01.036174    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3acf5ac04b96cfe3d2a2fa9a328eace9aa2d8b682408e3c0833552b5ac1e6667\": container with ID starting with 3acf5ac04b96cfe3d2a2fa9a328eace9aa2d8b682408e3c0833552b5ac1e6667 not found: ID does not exist" containerID="3acf5ac04b96cfe3d2a2fa9a328eace9aa2d8b682408e3c0833552b5ac1e6667"
Sep 16 12:34:01.036273 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:01.036212    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3acf5ac04b96cfe3d2a2fa9a328eace9aa2d8b682408e3c0833552b5ac1e6667"} err="failed to get container status \"3acf5ac04b96cfe3d2a2fa9a328eace9aa2d8b682408e3c0833552b5ac1e6667\": rpc error: code = NotFound desc = could not find container \"3acf5ac04b96cfe3d2a2fa9a328eace9aa2d8b682408e3c0833552b5ac1e6667\": container with ID starting with 3acf5ac04b96cfe3d2a2fa9a328eace9aa2d8b682408e3c0833552b5ac1e6667 not found: ID does not exist"
Sep 16 12:34:01.036273 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:01.036240    2579 scope.go:117] "RemoveContainer" containerID="c7f6d20ddf4275b3aa4a9a725da93409fbe273ae93cd796439740aea9be792ed"
Sep 16 12:34:01.036758 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:34:01.036731    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7f6d20ddf4275b3aa4a9a725da93409fbe273ae93cd796439740aea9be792ed\": container with ID starting with c7f6d20ddf4275b3aa4a9a725da93409fbe273ae93cd796439740aea9be792ed not found: ID does not exist" containerID="c7f6d20ddf4275b3aa4a9a725da93409fbe273ae93cd796439740aea9be792ed"
Sep 16 12:34:01.036870 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:01.036762    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f6d20ddf4275b3aa4a9a725da93409fbe273ae93cd796439740aea9be792ed"} err="failed to get container status \"c7f6d20ddf4275b3aa4a9a725da93409fbe273ae93cd796439740aea9be792ed\": rpc error: code = NotFound desc = could not find container \"c7f6d20ddf4275b3aa4a9a725da93409fbe273ae93cd796439740aea9be792ed\": container with ID starting with c7f6d20ddf4275b3aa4a9a725da93409fbe273ae93cd796439740aea9be792ed not found: ID does not exist"
Sep 16 12:34:01.039806 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:01.039781    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"]
Sep 16 12:34:01.044842 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:01.044785    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-51ae50-openshift-default-6f99749974-ghkd7"]
Sep 16 12:34:02.781581 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:02.781532    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33b4eb96-7651-48ab-9256-3614eeb10f6c" path="/var/lib/kubelet/pods/33b4eb96-7651-48ab-9256-3614eeb10f6c/volumes"
Sep 16 12:34:10.796967 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:10.796843    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:34:12.166976 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.166937    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7883-48fcf9"]
Sep 16 12:34:12.167354 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167316    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d2173d2-5881-4e76-8de6-548d486111db" containerName="main"
Sep 16 12:34:12.167354 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167327    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d2173d2-5881-4e76-8de6-548d486111db" containerName="main"
Sep 16 12:34:12.167354 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167337    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33b4eb96-7651-48ab-9256-3614eeb10f6c" containerName="istio-proxy"
Sep 16 12:34:12.167354 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167343    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="33b4eb96-7651-48ab-9256-3614eeb10f6c" containerName="istio-proxy"
Sep 16 12:34:12.167354 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167354    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c7be173-8ad3-42ce-ab28-c5f73f349438" containerName="maas-api"
Sep 16 12:34:12.167519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167359    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c7be173-8ad3-42ce-ab28-c5f73f349438" containerName="maas-api"
Sep 16 12:34:12.167519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167368    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaa9624f-72d7-4a15-abcc-ba19f761388a" containerName="istio-proxy"
Sep 16 12:34:12.167519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167373    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaa9624f-72d7-4a15-abcc-ba19f761388a" containerName="istio-proxy"
Sep 16 12:34:12.167519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167382    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33b4eb96-7651-48ab-9256-3614eeb10f6c" containerName="istio-proxy"
Sep 16 12:34:12.167519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167388    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="33b4eb96-7651-48ab-9256-3614eeb10f6c" containerName="istio-proxy"
Sep 16 12:34:12.167519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167398    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="787efc7d-197f-4c85-9266-7e38ffaf83db" containerName="maas-api"
Sep 16 12:34:12.167519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167404    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="787efc7d-197f-4c85-9266-7e38ffaf83db" containerName="maas-api"
Sep 16 12:34:12.167519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167431    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ade63ee-3184-46df-ada2-55a290bcbefc" containerName="istio-proxy"
Sep 16 12:34:12.167519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167437    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ade63ee-3184-46df-ada2-55a290bcbefc" containerName="istio-proxy"
Sep 16 12:34:12.167519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167444    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ade63ee-3184-46df-ada2-55a290bcbefc" containerName="istio-proxy"
Sep 16 12:34:12.167519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167450    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ade63ee-3184-46df-ada2-55a290bcbefc" containerName="istio-proxy"
Sep 16 12:34:12.167519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167456    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c511a6e4-dddc-4b5b-b7ce-11b31bb263ae" containerName="maas-api"
Sep 16 12:34:12.167519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167461    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c511a6e4-dddc-4b5b-b7ce-11b31bb263ae" containerName="maas-api"
Sep 16 12:34:12.167519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167467    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b97355ae-45e5-4812-b0b1-3abc6d7a0c69" containerName="main"
Sep 16 12:34:12.167519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167472    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b97355ae-45e5-4812-b0b1-3abc6d7a0c69" containerName="main"
Sep 16 12:34:12.167519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167479    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8cad60f-d2a0-4b30-a107-68f0bf623ae2" containerName="maas-api"
Sep 16 12:34:12.167519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167484    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8cad60f-d2a0-4b30-a107-68f0bf623ae2" containerName="maas-api"
Sep 16 12:34:12.168074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167539    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ade63ee-3184-46df-ada2-55a290bcbefc" containerName="istio-proxy"
Sep 16 12:34:12.168074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167547    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c7be173-8ad3-42ce-ab28-c5f73f349438" containerName="maas-api"
Sep 16 12:34:12.168074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167554    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b97355ae-45e5-4812-b0b1-3abc6d7a0c69" containerName="main"
Sep 16 12:34:12.168074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167560    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaa9624f-72d7-4a15-abcc-ba19f761388a" containerName="istio-proxy"
Sep 16 12:34:12.168074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167566    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="33b4eb96-7651-48ab-9256-3614eeb10f6c" containerName="istio-proxy"
Sep 16 12:34:12.168074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167572    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8cad60f-d2a0-4b30-a107-68f0bf623ae2" containerName="maas-api"
Sep 16 12:34:12.168074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167579    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c511a6e4-dddc-4b5b-b7ce-11b31bb263ae" containerName="maas-api"
Sep 16 12:34:12.168074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167585    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="787efc7d-197f-4c85-9266-7e38ffaf83db" containerName="maas-api"
Sep 16 12:34:12.168074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167591    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="33b4eb96-7651-48ab-9256-3614eeb10f6c" containerName="istio-proxy"
Sep 16 12:34:12.168074 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.167596    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d2173d2-5881-4e76-8de6-548d486111db" containerName="main"
Sep 16 12:34:12.172143 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.172126    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7883-48fcf9"
Sep 16 12:34:12.175018 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.174991    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6btc9\""
Sep 16 12:34:12.177355 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.177325    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7883-48fcf9"]
Sep 16 12:34:12.256714 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.256677    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f42d4\" (UniqueName: \"kubernetes.io/projected/9627b169-1a4e-448a-874b-1f22f2799f05-kube-api-access-f42d4\") pod \"test-curl-7883-48fcf9\" (UID: \"9627b169-1a4e-448a-874b-1f22f2799f05\") " pod="opendatahub/test-curl-7883-48fcf9"
Sep 16 12:34:12.357543 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.357504    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f42d4\" (UniqueName: \"kubernetes.io/projected/9627b169-1a4e-448a-874b-1f22f2799f05-kube-api-access-f42d4\") pod \"test-curl-7883-48fcf9\" (UID: \"9627b169-1a4e-448a-874b-1f22f2799f05\") " pod="opendatahub/test-curl-7883-48fcf9"
Sep 16 12:34:12.367131 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.367093    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f42d4\" (UniqueName: \"kubernetes.io/projected/9627b169-1a4e-448a-874b-1f22f2799f05-kube-api-access-f42d4\") pod \"test-curl-7883-48fcf9\" (UID: \"9627b169-1a4e-448a-874b-1f22f2799f05\") " pod="opendatahub/test-curl-7883-48fcf9"
Sep 16 12:34:12.483064 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.482971    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7883-48fcf9"
Sep 16 12:34:12.756995 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:12.756967    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7883-48fcf9"]
Sep 16 12:34:12.758717 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:34:12.758687    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9627b169_1a4e_448a_874b_1f22f2799f05.slice/crio-1823fef29a570396c84efb2f0696921d1d9cb30cd82ee0efcc0c739d28873d32 WatchSource:0}: Error finding container 1823fef29a570396c84efb2f0696921d1d9cb30cd82ee0efcc0c739d28873d32: Status 404 returned error can't find the container with id 1823fef29a570396c84efb2f0696921d1d9cb30cd82ee0efcc0c739d28873d32
Sep 16 12:34:13.092820 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:13.092787    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7883-48fcf9" event={"ID":"9627b169-1a4e-448a-874b-1f22f2799f05","Type":"ContainerStarted","Data":"cdf416403d46737f28f4798c37a30e2b0c0bc7bc871967d9ddf0bc3d1f27cd55"}
Sep 16 12:34:13.092820 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:13.092827    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7883-48fcf9" event={"ID":"9627b169-1a4e-448a-874b-1f22f2799f05","Type":"ContainerStarted","Data":"1823fef29a570396c84efb2f0696921d1d9cb30cd82ee0efcc0c739d28873d32"}
Sep 16 12:34:13.783258 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:13.783204    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7883-48fcf9" podStartSLOduration=1.783186153 podStartE2EDuration="1.783186153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:34:13.115574669 +0000 UTC m=+1846.816645995" watchObservedRunningTime="2026-09-16 12:34:13.783186153 +0000 UTC m=+1847.484257479"
Sep 16 12:34:13.784845 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:13.784824    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7883-48fcf9"]
Sep 16 12:34:13.791719 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:13.791690    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7883-48fcf9"]
Sep 16 12:34:14.098128 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:34:14.098099    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7883-48fcf9" container="test-curl-7883-48fcf9"
Sep 16 12:34:15.103703 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:15.103640    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7883-48fcf9" podUID="9627b169-1a4e-448a-874b-1f22f2799f05" containerName="test-curl-7883-48fcf9" containerID="cri-o://cdf416403d46737f28f4798c37a30e2b0c0bc7bc871967d9ddf0bc3d1f27cd55" gracePeriod=2
Sep 16 12:34:17.290411 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:17.290382    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7883-48fcf9"
Sep 16 12:34:17.405061 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:17.404970    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f42d4\" (UniqueName: \"kubernetes.io/projected/9627b169-1a4e-448a-874b-1f22f2799f05-kube-api-access-f42d4\") pod \"9627b169-1a4e-448a-874b-1f22f2799f05\" (UID: \"9627b169-1a4e-448a-874b-1f22f2799f05\") "
Sep 16 12:34:17.407022 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:17.406999    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9627b169-1a4e-448a-874b-1f22f2799f05-kube-api-access-f42d4" (OuterVolumeSpecName: "kube-api-access-f42d4") pod "9627b169-1a4e-448a-874b-1f22f2799f05" (UID: "9627b169-1a4e-448a-874b-1f22f2799f05"). InnerVolumeSpecName "kube-api-access-f42d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:34:17.505851 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:17.505815    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f42d4\" (UniqueName: \"kubernetes.io/projected/9627b169-1a4e-448a-874b-1f22f2799f05-kube-api-access-f42d4\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:34:18.122176 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:18.122143    2579 generic.go:356] "Generic (PLEG): container finished" podID="9627b169-1a4e-448a-874b-1f22f2799f05" containerID="cdf416403d46737f28f4798c37a30e2b0c0bc7bc871967d9ddf0bc3d1f27cd55" exitCode=137
Sep 16 12:34:18.122372 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:18.122190    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7883-48fcf9"
Sep 16 12:34:18.122372 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:18.122243    2579 scope.go:117] "RemoveContainer" containerID="cdf416403d46737f28f4798c37a30e2b0c0bc7bc871967d9ddf0bc3d1f27cd55"
Sep 16 12:34:18.132671 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:18.132638    2579 scope.go:117] "RemoveContainer" containerID="cdf416403d46737f28f4798c37a30e2b0c0bc7bc871967d9ddf0bc3d1f27cd55"
Sep 16 12:34:18.132938 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:34:18.132919    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf416403d46737f28f4798c37a30e2b0c0bc7bc871967d9ddf0bc3d1f27cd55\": container with ID starting with cdf416403d46737f28f4798c37a30e2b0c0bc7bc871967d9ddf0bc3d1f27cd55 not found: ID does not exist" containerID="cdf416403d46737f28f4798c37a30e2b0c0bc7bc871967d9ddf0bc3d1f27cd55"
Sep 16 12:34:18.133000 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:18.132947    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf416403d46737f28f4798c37a30e2b0c0bc7bc871967d9ddf0bc3d1f27cd55"} err="failed to get container status \"cdf416403d46737f28f4798c37a30e2b0c0bc7bc871967d9ddf0bc3d1f27cd55\": rpc error: code = NotFound desc = could not find container \"cdf416403d46737f28f4798c37a30e2b0c0bc7bc871967d9ddf0bc3d1f27cd55\": container with ID starting with cdf416403d46737f28f4798c37a30e2b0c0bc7bc871967d9ddf0bc3d1f27cd55 not found: ID does not exist"
Sep 16 12:34:18.137498 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:34:18.137469    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7883-48fcf9\" is forbidden: User \"system:node:ip-10-0-137-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-30.ec2.internal' and this object" podUID="9627b169-1a4e-448a-874b-1f22f2799f05" pod="opendatahub/test-curl-7883-48fcf9"
Sep 16 12:34:18.781269 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:18.781234    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9627b169-1a4e-448a-874b-1f22f2799f05" path="/var/lib/kubelet/pods/9627b169-1a4e-448a-874b-1f22f2799f05/volumes"
Sep 16 12:34:24.283901 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:24.283868    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:34:36.322553 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:36.322518    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:34:59.873700 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:34:59.873652    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:35:29.179021 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:29.178984    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:35:37.309692 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:37.309645    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7883-df107a"]
Sep 16 12:35:37.310219 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:37.310110    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9627b169-1a4e-448a-874b-1f22f2799f05" containerName="test-curl-7883-48fcf9"
Sep 16 12:35:37.310219 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:37.310123    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9627b169-1a4e-448a-874b-1f22f2799f05" containerName="test-curl-7883-48fcf9"
Sep 16 12:35:37.310219 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:37.310155    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaa9624f-72d7-4a15-abcc-ba19f761388a" containerName="istio-proxy"
Sep 16 12:35:37.310219 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:37.310161    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaa9624f-72d7-4a15-abcc-ba19f761388a" containerName="istio-proxy"
Sep 16 12:35:37.310430 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:37.310232    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9627b169-1a4e-448a-874b-1f22f2799f05" containerName="test-curl-7883-48fcf9"
Sep 16 12:35:37.310430 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:37.310239    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ade63ee-3184-46df-ada2-55a290bcbefc" containerName="istio-proxy"
Sep 16 12:35:37.310430 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:37.310247    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaa9624f-72d7-4a15-abcc-ba19f761388a" containerName="istio-proxy"
Sep 16 12:35:37.313832 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:37.313808    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7883-df107a"
Sep 16 12:35:37.316629 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:37.316590    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6btc9\""
Sep 16 12:35:37.317439 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:37.317412    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7883-df107a"]
Sep 16 12:35:37.345465 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:37.345435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g85kk\" (UniqueName: \"kubernetes.io/projected/509d7fca-10d7-4cc0-ad0c-3d0c351bff9c-kube-api-access-g85kk\") pod \"test-curl-7883-df107a\" (UID: \"509d7fca-10d7-4cc0-ad0c-3d0c351bff9c\") " pod="opendatahub/test-curl-7883-df107a"
Sep 16 12:35:37.446184 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:37.446148    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g85kk\" (UniqueName: \"kubernetes.io/projected/509d7fca-10d7-4cc0-ad0c-3d0c351bff9c-kube-api-access-g85kk\") pod \"test-curl-7883-df107a\" (UID: \"509d7fca-10d7-4cc0-ad0c-3d0c351bff9c\") " pod="opendatahub/test-curl-7883-df107a"
Sep 16 12:35:37.456545 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:37.456506    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g85kk\" (UniqueName: \"kubernetes.io/projected/509d7fca-10d7-4cc0-ad0c-3d0c351bff9c-kube-api-access-g85kk\") pod \"test-curl-7883-df107a\" (UID: \"509d7fca-10d7-4cc0-ad0c-3d0c351bff9c\") " pod="opendatahub/test-curl-7883-df107a"
Sep 16 12:35:37.625135 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:37.625098    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7883-df107a"
Sep 16 12:35:37.895376 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:37.895349    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7883-df107a"]
Sep 16 12:35:37.897278 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:35:37.897240    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod509d7fca_10d7_4cc0_ad0c_3d0c351bff9c.slice/crio-16b1a38f23cdf7a9d267b445bb8410487ff098a7616eccdbc3fa36c7c6bae11a WatchSource:0}: Error finding container 16b1a38f23cdf7a9d267b445bb8410487ff098a7616eccdbc3fa36c7c6bae11a: Status 404 returned error can't find the container with id 16b1a38f23cdf7a9d267b445bb8410487ff098a7616eccdbc3fa36c7c6bae11a
Sep 16 12:35:38.581491 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:38.581450    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7883-df107a" event={"ID":"509d7fca-10d7-4cc0-ad0c-3d0c351bff9c","Type":"ContainerStarted","Data":"0737683f8bcb127ca459d3ee79715e2b661c0a90c1b3d1e28ee96504dae3e60a"}
Sep 16 12:35:38.581922 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:38.581504    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7883-df107a" event={"ID":"509d7fca-10d7-4cc0-ad0c-3d0c351bff9c","Type":"ContainerStarted","Data":"16b1a38f23cdf7a9d267b445bb8410487ff098a7616eccdbc3fa36c7c6bae11a"}
Sep 16 12:35:38.603218 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:38.603168    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7883-df107a" podStartSLOduration=1.6031526729999999 podStartE2EDuration="1.603152673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:35:38.599801914 +0000 UTC m=+1932.300873239" watchObservedRunningTime="2026-09-16 12:35:38.603152673 +0000 UTC m=+1932.304223999"
Sep 16 12:35:39.209883 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:39.209853    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7883-df107a"]
Sep 16 12:35:39.213835 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:39.213805    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7883-df107a"]
Sep 16 12:35:39.327798 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:39.327705    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7883-3a1670"]
Sep 16 12:35:39.328201 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:39.328180    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="509d7fca-10d7-4cc0-ad0c-3d0c351bff9c" containerName="test-curl-7883-df107a"
Sep 16 12:35:39.328295 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:39.328202    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="509d7fca-10d7-4cc0-ad0c-3d0c351bff9c" containerName="test-curl-7883-df107a"
Sep 16 12:35:39.328358 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:39.328340    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="509d7fca-10d7-4cc0-ad0c-3d0c351bff9c" containerName="test-curl-7883-df107a"
Sep 16 12:35:39.331443 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:39.331422    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7883-3a1670"
Sep 16 12:35:39.335949 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:39.335924    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7883-3a1670"]
Sep 16 12:35:39.364286 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:39.364259    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhsb6\" (UniqueName: \"kubernetes.io/projected/e5d3a426-2668-48f7-b9c1-da82d17d39cc-kube-api-access-rhsb6\") pod \"test-curl-7883-3a1670\" (UID: \"e5d3a426-2668-48f7-b9c1-da82d17d39cc\") " pod="opendatahub/test-curl-7883-3a1670"
Sep 16 12:35:39.465181 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:39.465136    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhsb6\" (UniqueName: \"kubernetes.io/projected/e5d3a426-2668-48f7-b9c1-da82d17d39cc-kube-api-access-rhsb6\") pod \"test-curl-7883-3a1670\" (UID: \"e5d3a426-2668-48f7-b9c1-da82d17d39cc\") " pod="opendatahub/test-curl-7883-3a1670"
Sep 16 12:35:39.473669 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:39.473640    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhsb6\" (UniqueName: \"kubernetes.io/projected/e5d3a426-2668-48f7-b9c1-da82d17d39cc-kube-api-access-rhsb6\") pod \"test-curl-7883-3a1670\" (UID: \"e5d3a426-2668-48f7-b9c1-da82d17d39cc\") " pod="opendatahub/test-curl-7883-3a1670"
Sep 16 12:35:39.587706 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:35:39.587681    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7883-df107a" container="test-curl-7883-df107a"
Sep 16 12:35:39.643418 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:39.643389    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7883-3a1670"
Sep 16 12:35:39.913750 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:39.913715    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7883-3a1670"]
Sep 16 12:35:39.914817 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:35:39.914789    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5d3a426_2668_48f7_b9c1_da82d17d39cc.slice/crio-2d518a7272b0a773959b7a6cb38999f37774174730708bccf692bf784f0c23a3 WatchSource:0}: Error finding container 2d518a7272b0a773959b7a6cb38999f37774174730708bccf692bf784f0c23a3: Status 404 returned error can't find the container with id 2d518a7272b0a773959b7a6cb38999f37774174730708bccf692bf784f0c23a3
Sep 16 12:35:40.595970 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:40.595932    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7883-3a1670" event={"ID":"e5d3a426-2668-48f7-b9c1-da82d17d39cc","Type":"ContainerStarted","Data":"1c07273d45f94c4384f8dcc92e7b1776532ce09edeee5fa0b04019222018b70b"}
Sep 16 12:35:40.596436 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:40.595979    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7883-3a1670" event={"ID":"e5d3a426-2668-48f7-b9c1-da82d17d39cc","Type":"ContainerStarted","Data":"2d518a7272b0a773959b7a6cb38999f37774174730708bccf692bf784f0c23a3"}
Sep 16 12:35:40.596436 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:40.596002    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7883-df107a" podUID="509d7fca-10d7-4cc0-ad0c-3d0c351bff9c" containerName="test-curl-7883-df107a" containerID="cri-o://0737683f8bcb127ca459d3ee79715e2b661c0a90c1b3d1e28ee96504dae3e60a" gracePeriod=2
Sep 16 12:35:40.612910 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:40.612858    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7883-3a1670" podStartSLOduration=1.6128431600000002 podStartE2EDuration="1.61284316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:35:40.611470607 +0000 UTC m=+1934.312541934" watchObservedRunningTime="2026-09-16 12:35:40.61284316 +0000 UTC m=+1934.313914486"
Sep 16 12:35:40.613578 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:35:40.613553    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7883-df107a\" is forbidden: User \"system:node:ip-10-0-137-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-30.ec2.internal' and this object" podUID="509d7fca-10d7-4cc0-ad0c-3d0c351bff9c" pod="opendatahub/test-curl-7883-df107a"
Sep 16 12:35:40.615506 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:35:40.615485    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7883-df107a\" is forbidden: User \"system:node:ip-10-0-137-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-30.ec2.internal' and this object" podUID="509d7fca-10d7-4cc0-ad0c-3d0c351bff9c" pod="opendatahub/test-curl-7883-df107a"
Sep 16 12:35:41.200833 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:41.200793    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7883-3a1670"]
Sep 16 12:35:41.204479 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:41.204450    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7883-3a1670"]
Sep 16 12:35:41.602241 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:35:41.602208    2579 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7883-3a1670" container="test-curl-7883-3a1670"
Sep 16 12:35:42.273276 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:42.273244    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:35:42.607410 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:42.607375    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7883-3a1670" podUID="e5d3a426-2668-48f7-b9c1-da82d17d39cc" containerName="test-curl-7883-3a1670" containerID="cri-o://1c07273d45f94c4384f8dcc92e7b1776532ce09edeee5fa0b04019222018b70b" gracePeriod=2
Sep 16 12:35:42.610102 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:35:42.610072    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7883-3a1670\" is forbidden: User \"system:node:ip-10-0-137-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-30.ec2.internal' and this object" podUID="e5d3a426-2668-48f7-b9c1-da82d17d39cc" pod="opendatahub/test-curl-7883-3a1670"
Sep 16 12:35:42.778791 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:42.778764    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7883-df107a"
Sep 16 12:35:42.781549 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:35:42.781517    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7883-3a1670\" is forbidden: User \"system:node:ip-10-0-137-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-30.ec2.internal' and this object" podUID="e5d3a426-2668-48f7-b9c1-da82d17d39cc" pod="opendatahub/test-curl-7883-3a1670"
Sep 16 12:35:42.797610 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:42.797574    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g85kk\" (UniqueName: \"kubernetes.io/projected/509d7fca-10d7-4cc0-ad0c-3d0c351bff9c-kube-api-access-g85kk\") pod \"509d7fca-10d7-4cc0-ad0c-3d0c351bff9c\" (UID: \"509d7fca-10d7-4cc0-ad0c-3d0c351bff9c\") "
Sep 16 12:35:42.799670 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:42.799624    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/509d7fca-10d7-4cc0-ad0c-3d0c351bff9c-kube-api-access-g85kk" (OuterVolumeSpecName: "kube-api-access-g85kk") pod "509d7fca-10d7-4cc0-ad0c-3d0c351bff9c" (UID: "509d7fca-10d7-4cc0-ad0c-3d0c351bff9c"). InnerVolumeSpecName "kube-api-access-g85kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:35:42.898450 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:42.898355    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g85kk\" (UniqueName: \"kubernetes.io/projected/509d7fca-10d7-4cc0-ad0c-3d0c351bff9c-kube-api-access-g85kk\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:35:43.613961 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:43.613927    2579 generic.go:356] "Generic (PLEG): container finished" podID="509d7fca-10d7-4cc0-ad0c-3d0c351bff9c" containerID="0737683f8bcb127ca459d3ee79715e2b661c0a90c1b3d1e28ee96504dae3e60a" exitCode=137
Sep 16 12:35:43.614344 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:43.613970    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7883-df107a"
Sep 16 12:35:43.614344 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:43.614027    2579 scope.go:117] "RemoveContainer" containerID="0737683f8bcb127ca459d3ee79715e2b661c0a90c1b3d1e28ee96504dae3e60a"
Sep 16 12:35:43.623584 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:43.623566    2579 scope.go:117] "RemoveContainer" containerID="0737683f8bcb127ca459d3ee79715e2b661c0a90c1b3d1e28ee96504dae3e60a"
Sep 16 12:35:43.623884 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:35:43.623863    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0737683f8bcb127ca459d3ee79715e2b661c0a90c1b3d1e28ee96504dae3e60a\": container with ID starting with 0737683f8bcb127ca459d3ee79715e2b661c0a90c1b3d1e28ee96504dae3e60a not found: ID does not exist" containerID="0737683f8bcb127ca459d3ee79715e2b661c0a90c1b3d1e28ee96504dae3e60a"
Sep 16 12:35:43.623953 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:43.623896    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0737683f8bcb127ca459d3ee79715e2b661c0a90c1b3d1e28ee96504dae3e60a"} err="failed to get container status \"0737683f8bcb127ca459d3ee79715e2b661c0a90c1b3d1e28ee96504dae3e60a\": rpc error: code = NotFound desc = could not find container \"0737683f8bcb127ca459d3ee79715e2b661c0a90c1b3d1e28ee96504dae3e60a\": container with ID starting with 0737683f8bcb127ca459d3ee79715e2b661c0a90c1b3d1e28ee96504dae3e60a not found: ID does not exist"
Sep 16 12:35:44.781473 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:44.781442    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="509d7fca-10d7-4cc0-ad0c-3d0c351bff9c" path="/var/lib/kubelet/pods/509d7fca-10d7-4cc0-ad0c-3d0c351bff9c/volumes"
Sep 16 12:35:44.791586 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:44.791568    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7883-3a1670"
Sep 16 12:35:44.818345 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:44.818315    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhsb6\" (UniqueName: \"kubernetes.io/projected/e5d3a426-2668-48f7-b9c1-da82d17d39cc-kube-api-access-rhsb6\") pod \"e5d3a426-2668-48f7-b9c1-da82d17d39cc\" (UID: \"e5d3a426-2668-48f7-b9c1-da82d17d39cc\") "
Sep 16 12:35:44.820308 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:44.820282    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d3a426-2668-48f7-b9c1-da82d17d39cc-kube-api-access-rhsb6" (OuterVolumeSpecName: "kube-api-access-rhsb6") pod "e5d3a426-2668-48f7-b9c1-da82d17d39cc" (UID: "e5d3a426-2668-48f7-b9c1-da82d17d39cc"). InnerVolumeSpecName "kube-api-access-rhsb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:35:44.919836 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:44.919746    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhsb6\" (UniqueName: \"kubernetes.io/projected/e5d3a426-2668-48f7-b9c1-da82d17d39cc-kube-api-access-rhsb6\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:35:45.627905 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:45.627866    2579 generic.go:356] "Generic (PLEG): container finished" podID="e5d3a426-2668-48f7-b9c1-da82d17d39cc" containerID="1c07273d45f94c4384f8dcc92e7b1776532ce09edeee5fa0b04019222018b70b" exitCode=137
Sep 16 12:35:45.628109 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:45.627914    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7883-3a1670"
Sep 16 12:35:45.628109 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:45.627952    2579 scope.go:117] "RemoveContainer" containerID="1c07273d45f94c4384f8dcc92e7b1776532ce09edeee5fa0b04019222018b70b"
Sep 16 12:35:45.637173 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:45.637150    2579 scope.go:117] "RemoveContainer" containerID="1c07273d45f94c4384f8dcc92e7b1776532ce09edeee5fa0b04019222018b70b"
Sep 16 12:35:45.637427 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:35:45.637409    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c07273d45f94c4384f8dcc92e7b1776532ce09edeee5fa0b04019222018b70b\": container with ID starting with 1c07273d45f94c4384f8dcc92e7b1776532ce09edeee5fa0b04019222018b70b not found: ID does not exist" containerID="1c07273d45f94c4384f8dcc92e7b1776532ce09edeee5fa0b04019222018b70b"
Sep 16 12:35:45.637491 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:45.637432    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c07273d45f94c4384f8dcc92e7b1776532ce09edeee5fa0b04019222018b70b"} err="failed to get container status \"1c07273d45f94c4384f8dcc92e7b1776532ce09edeee5fa0b04019222018b70b\": rpc error: code = NotFound desc = could not find container \"1c07273d45f94c4384f8dcc92e7b1776532ce09edeee5fa0b04019222018b70b\": container with ID starting with 1c07273d45f94c4384f8dcc92e7b1776532ce09edeee5fa0b04019222018b70b not found: ID does not exist"
Sep 16 12:35:45.638872 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:35:45.638851    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7883-3a1670\" is forbidden: User \"system:node:ip-10-0-137-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-30.ec2.internal' and this object" podUID="e5d3a426-2668-48f7-b9c1-da82d17d39cc" pod="opendatahub/test-curl-7883-3a1670"
Sep 16 12:35:46.779703 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:35:46.779640    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7883-3a1670\" is forbidden: User \"system:node:ip-10-0-137-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-30.ec2.internal' and this object" podUID="e5d3a426-2668-48f7-b9c1-da82d17d39cc" pod="opendatahub/test-curl-7883-3a1670"
Sep 16 12:35:46.781055 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:46.781033    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5d3a426-2668-48f7-b9c1-da82d17d39cc" path="/var/lib/kubelet/pods/e5d3a426-2668-48f7-b9c1-da82d17d39cc/volumes"
Sep 16 12:35:50.674927 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:35:50.674892    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:36:19.089465 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:36:19.089384    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:36:27.112284 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:36:27.112253    2579 scope.go:117] "RemoveContainer" containerID="c9f2996b4cc05ede55a05135b31bb8a791300f7f186659ed7bbfeaea828ea201"
Sep 16 12:36:40.180456 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:36:40.180421    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:36:57.177015 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:36:57.176976    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:37:06.084719 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:06.084683    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:37:16.295631 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:16.295596    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:37:28.560104 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:28.560064    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:37:48.830623 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:48.830588    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh"]
Sep 16 12:37:48.831024 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:48.831009    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5d3a426-2668-48f7-b9c1-da82d17d39cc" containerName="test-curl-7883-3a1670"
Sep 16 12:37:48.831068 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:48.831025    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5d3a426-2668-48f7-b9c1-da82d17d39cc" containerName="test-curl-7883-3a1670"
Sep 16 12:37:48.831110 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:48.831100    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5d3a426-2668-48f7-b9c1-da82d17d39cc" containerName="test-curl-7883-3a1670"
Sep 16 12:37:48.834141 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:48.834116    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh"
Sep 16 12:37:48.836863 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:48.836837    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-89b7ca-serving-cert\""
Sep 16 12:37:48.850005 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:48.849979    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh"]
Sep 16 12:37:48.889621 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:48.889590    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b79691e9-908e-49bb-af57-00b6ca7c17c4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh\" (UID: \"b79691e9-908e-49bb-af57-00b6ca7c17c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh"
Sep 16 12:37:48.889621 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:48.889623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b79691e9-908e-49bb-af57-00b6ca7c17c4-metrics-certs\") pod \"maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh\" (UID: \"b79691e9-908e-49bb-af57-00b6ca7c17c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh"
Sep 16 12:37:48.889841 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:48.889825    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b79691e9-908e-49bb-af57-00b6ca7c17c4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh\" (UID: \"b79691e9-908e-49bb-af57-00b6ca7c17c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh"
Sep 16 12:37:48.889937 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:48.889920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhvbq\" (UniqueName: \"kubernetes.io/projected/b79691e9-908e-49bb-af57-00b6ca7c17c4-kube-api-access-fhvbq\") pod \"maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh\" (UID: \"b79691e9-908e-49bb-af57-00b6ca7c17c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh"
Sep 16 12:37:48.991164 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:48.991085    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b79691e9-908e-49bb-af57-00b6ca7c17c4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh\" (UID: \"b79691e9-908e-49bb-af57-00b6ca7c17c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh"
Sep 16 12:37:48.991164 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:48.991140    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhvbq\" (UniqueName: \"kubernetes.io/projected/b79691e9-908e-49bb-af57-00b6ca7c17c4-kube-api-access-fhvbq\") pod \"maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh\" (UID: \"b79691e9-908e-49bb-af57-00b6ca7c17c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh"
Sep 16 12:37:48.991388 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:48.991210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b79691e9-908e-49bb-af57-00b6ca7c17c4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh\" (UID: \"b79691e9-908e-49bb-af57-00b6ca7c17c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh"
Sep 16 12:37:48.991388 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:48.991325    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b79691e9-908e-49bb-af57-00b6ca7c17c4-metrics-certs\") pod \"maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh\" (UID: \"b79691e9-908e-49bb-af57-00b6ca7c17c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh"
Sep 16 12:37:48.991914 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:48.991892    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b79691e9-908e-49bb-af57-00b6ca7c17c4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh\" (UID: \"b79691e9-908e-49bb-af57-00b6ca7c17c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh"
Sep 16 12:37:48.993571 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:48.993548    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b79691e9-908e-49bb-af57-00b6ca7c17c4-metrics-certs\") pod \"maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh\" (UID: \"b79691e9-908e-49bb-af57-00b6ca7c17c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh"
Sep 16 12:37:48.993688 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:48.993627    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b79691e9-908e-49bb-af57-00b6ca7c17c4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh\" (UID: \"b79691e9-908e-49bb-af57-00b6ca7c17c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh"
Sep 16 12:37:49.002491 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:49.001541    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhvbq\" (UniqueName: \"kubernetes.io/projected/b79691e9-908e-49bb-af57-00b6ca7c17c4-kube-api-access-fhvbq\") pod \"maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh\" (UID: \"b79691e9-908e-49bb-af57-00b6ca7c17c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh"
Sep 16 12:37:49.144556 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:49.144520    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh"
Sep 16 12:37:49.425971 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:49.425941    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh"]
Sep 16 12:37:49.428168 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:37:49.428136    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb79691e9_908e_49bb_af57_00b6ca7c17c4.slice/crio-f55fc25eefb536d46d7f28529c39b828f5b21b760e466e42f05b8550d2b5bed5 WatchSource:0}: Error finding container f55fc25eefb536d46d7f28529c39b828f5b21b760e466e42f05b8550d2b5bed5: Status 404 returned error can't find the container with id f55fc25eefb536d46d7f28529c39b828f5b21b760e466e42f05b8550d2b5bed5
Sep 16 12:37:49.429482 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:49.429463    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:37:50.324369 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:50.324328    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh" event={"ID":"b79691e9-908e-49bb-af57-00b6ca7c17c4","Type":"ContainerStarted","Data":"f6c30d135266b63572b5f1ba21ef5b038b9a3f37b3c155e380719f6e1eb1c57d"}
Sep 16 12:37:50.324906 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:50.324399    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh" event={"ID":"b79691e9-908e-49bb-af57-00b6ca7c17c4","Type":"ContainerStarted","Data":"f55fc25eefb536d46d7f28529c39b828f5b21b760e466e42f05b8550d2b5bed5"}
Sep 16 12:37:50.324906 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:50.324431    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh"
Sep 16 12:37:50.341986 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:50.341938    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh" podStartSLOduration=1.644815769 podStartE2EDuration="2.341921127s" totalImagesPullingTime="697.105358ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:37:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:37:50.340939586 +0000 UTC m=+2064.042010949" watchObservedRunningTime="2026-09-16 12:37:50.341921127 +0000 UTC m=+2064.042992452"
Sep 16 12:37:53.669573 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:53.669534    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:37:56.334954 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:37:56.334925    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh"
Sep 16 12:38:32.564242 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:38:32.564206    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:38:50.573158 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:38:50.573116    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:39:04.820972 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:04.820939    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp"]
Sep 16 12:39:04.824717 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:04.824692    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp"
Sep 16 12:39:04.827418 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:04.827393    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-b03a3f-serving-cert\""
Sep 16 12:39:04.837712 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:04.837687    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp"]
Sep 16 12:39:04.896913 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:04.896881    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a6b52b6-d72f-4224-81c3-fb5fcb142689-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp\" (UID: \"9a6b52b6-d72f-4224-81c3-fb5fcb142689\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp"
Sep 16 12:39:04.897088 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:04.896957    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bjvt\" (UniqueName: \"kubernetes.io/projected/9a6b52b6-d72f-4224-81c3-fb5fcb142689-kube-api-access-5bjvt\") pod \"maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp\" (UID: \"9a6b52b6-d72f-4224-81c3-fb5fcb142689\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp"
Sep 16 12:39:04.897088 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:04.896983    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a6b52b6-d72f-4224-81c3-fb5fcb142689-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp\" (UID: \"9a6b52b6-d72f-4224-81c3-fb5fcb142689\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp"
Sep 16 12:39:04.897088 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:04.897023    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a6b52b6-d72f-4224-81c3-fb5fcb142689-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp\" (UID: \"9a6b52b6-d72f-4224-81c3-fb5fcb142689\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp"
Sep 16 12:39:04.997761 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:04.997728    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a6b52b6-d72f-4224-81c3-fb5fcb142689-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp\" (UID: \"9a6b52b6-d72f-4224-81c3-fb5fcb142689\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp"
Sep 16 12:39:04.997946 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:04.997777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a6b52b6-d72f-4224-81c3-fb5fcb142689-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp\" (UID: \"9a6b52b6-d72f-4224-81c3-fb5fcb142689\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp"
Sep 16 12:39:04.997946 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:04.997833    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a6b52b6-d72f-4224-81c3-fb5fcb142689-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp\" (UID: \"9a6b52b6-d72f-4224-81c3-fb5fcb142689\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp"
Sep 16 12:39:04.998071 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:04.997966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bjvt\" (UniqueName: \"kubernetes.io/projected/9a6b52b6-d72f-4224-81c3-fb5fcb142689-kube-api-access-5bjvt\") pod \"maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp\" (UID: \"9a6b52b6-d72f-4224-81c3-fb5fcb142689\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp"
Sep 16 12:39:04.998585 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:04.998561    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a6b52b6-d72f-4224-81c3-fb5fcb142689-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp\" (UID: \"9a6b52b6-d72f-4224-81c3-fb5fcb142689\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp"
Sep 16 12:39:05.000218 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:05.000190    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a6b52b6-d72f-4224-81c3-fb5fcb142689-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp\" (UID: \"9a6b52b6-d72f-4224-81c3-fb5fcb142689\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp"
Sep 16 12:39:05.000334 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:05.000289    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a6b52b6-d72f-4224-81c3-fb5fcb142689-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp\" (UID: \"9a6b52b6-d72f-4224-81c3-fb5fcb142689\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp"
Sep 16 12:39:05.007522 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:05.007494    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bjvt\" (UniqueName: \"kubernetes.io/projected/9a6b52b6-d72f-4224-81c3-fb5fcb142689-kube-api-access-5bjvt\") pod \"maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp\" (UID: \"9a6b52b6-d72f-4224-81c3-fb5fcb142689\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp"
Sep 16 12:39:05.136246 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:05.136214    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp"
Sep 16 12:39:05.416383 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:39:05.416349    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6b52b6_d72f_4224_81c3_fb5fcb142689.slice/crio-5478f9d3facaca003952e8ea9caafb04b0cf39c7814b497c0223b9f696a48236 WatchSource:0}: Error finding container 5478f9d3facaca003952e8ea9caafb04b0cf39c7814b497c0223b9f696a48236: Status 404 returned error can't find the container with id 5478f9d3facaca003952e8ea9caafb04b0cf39c7814b497c0223b9f696a48236
Sep 16 12:39:05.417717 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:05.417689    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp"]
Sep 16 12:39:05.760594 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:05.760508    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp" event={"ID":"9a6b52b6-d72f-4224-81c3-fb5fcb142689","Type":"ContainerStarted","Data":"5478f9d3facaca003952e8ea9caafb04b0cf39c7814b497c0223b9f696a48236"}
Sep 16 12:39:06.780935 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:06.780895    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp" event={"ID":"9a6b52b6-d72f-4224-81c3-fb5fcb142689","Type":"ContainerStarted","Data":"e49c0f315fde8fdae62c598637f39581d5b2b44b12219de31ab5be262a233e2b"}
Sep 16 12:39:06.781316 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:06.780948    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp"
Sep 16 12:39:06.798443 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:06.798392    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp" podStartSLOduration=2.319346875 podStartE2EDuration="2.798378216s" totalImagesPullingTime="479.031341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:39:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:39:06.79710284 +0000 UTC m=+2140.498174189" watchObservedRunningTime="2026-09-16 12:39:06.798378216 +0000 UTC m=+2140.499449542"
Sep 16 12:39:09.970194 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:09.970158    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:39:12.785703 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:12.785650    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp"
Sep 16 12:39:43.062976 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:43.062945    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp"]
Sep 16 12:39:43.063604 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:43.063161    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp" podUID="9a6b52b6-d72f-4224-81c3-fb5fcb142689" containerName="maas-api" containerID="cri-o://e49c0f315fde8fdae62c598637f39581d5b2b44b12219de31ab5be262a233e2b" gracePeriod=30
Sep 16 12:39:43.323221 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:43.323154    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp"
Sep 16 12:39:43.369944 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:43.369907    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a6b52b6-d72f-4224-81c3-fb5fcb142689-maas-api-tls\") pod \"9a6b52b6-d72f-4224-81c3-fb5fcb142689\" (UID: \"9a6b52b6-d72f-4224-81c3-fb5fcb142689\") "
Sep 16 12:39:43.370111 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:43.369982    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a6b52b6-d72f-4224-81c3-fb5fcb142689-metrics-certs\") pod \"9a6b52b6-d72f-4224-81c3-fb5fcb142689\" (UID: \"9a6b52b6-d72f-4224-81c3-fb5fcb142689\") "
Sep 16 12:39:43.370111 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:43.370009    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a6b52b6-d72f-4224-81c3-fb5fcb142689-openshift-service-ca\") pod \"9a6b52b6-d72f-4224-81c3-fb5fcb142689\" (UID: \"9a6b52b6-d72f-4224-81c3-fb5fcb142689\") "
Sep 16 12:39:43.370111 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:43.370074    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bjvt\" (UniqueName: \"kubernetes.io/projected/9a6b52b6-d72f-4224-81c3-fb5fcb142689-kube-api-access-5bjvt\") pod \"9a6b52b6-d72f-4224-81c3-fb5fcb142689\" (UID: \"9a6b52b6-d72f-4224-81c3-fb5fcb142689\") "
Sep 16 12:39:43.370451 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:43.370417    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a6b52b6-d72f-4224-81c3-fb5fcb142689-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9a6b52b6-d72f-4224-81c3-fb5fcb142689" (UID: "9a6b52b6-d72f-4224-81c3-fb5fcb142689"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:39:43.372260 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:43.372221    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a6b52b6-d72f-4224-81c3-fb5fcb142689-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9a6b52b6-d72f-4224-81c3-fb5fcb142689" (UID: "9a6b52b6-d72f-4224-81c3-fb5fcb142689"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:39:43.372260 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:43.372217    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6b52b6-d72f-4224-81c3-fb5fcb142689-kube-api-access-5bjvt" (OuterVolumeSpecName: "kube-api-access-5bjvt") pod "9a6b52b6-d72f-4224-81c3-fb5fcb142689" (UID: "9a6b52b6-d72f-4224-81c3-fb5fcb142689"). InnerVolumeSpecName "kube-api-access-5bjvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:39:43.372439 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:43.372280    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a6b52b6-d72f-4224-81c3-fb5fcb142689-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9a6b52b6-d72f-4224-81c3-fb5fcb142689" (UID: "9a6b52b6-d72f-4224-81c3-fb5fcb142689"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:39:43.471229 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:43.471188    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a6b52b6-d72f-4224-81c3-fb5fcb142689-metrics-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:39:43.471229 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:43.471219    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a6b52b6-d72f-4224-81c3-fb5fcb142689-openshift-service-ca\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:39:43.471229 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:43.471231    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bjvt\" (UniqueName: \"kubernetes.io/projected/9a6b52b6-d72f-4224-81c3-fb5fcb142689-kube-api-access-5bjvt\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:39:43.471546 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:43.471240    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a6b52b6-d72f-4224-81c3-fb5fcb142689-maas-api-tls\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:39:43.985904 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:43.985870    2579 generic.go:356] "Generic (PLEG): container finished" podID="9a6b52b6-d72f-4224-81c3-fb5fcb142689" containerID="e49c0f315fde8fdae62c598637f39581d5b2b44b12219de31ab5be262a233e2b" exitCode=0
Sep 16 12:39:43.986052 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:43.985912    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp"
Sep 16 12:39:43.986052 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:43.985954    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp" event={"ID":"9a6b52b6-d72f-4224-81c3-fb5fcb142689","Type":"ContainerDied","Data":"e49c0f315fde8fdae62c598637f39581d5b2b44b12219de31ab5be262a233e2b"}
Sep 16 12:39:43.986052 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:43.986008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp" event={"ID":"9a6b52b6-d72f-4224-81c3-fb5fcb142689","Type":"ContainerDied","Data":"5478f9d3facaca003952e8ea9caafb04b0cf39c7814b497c0223b9f696a48236"}
Sep 16 12:39:43.986052 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:43.986031    2579 scope.go:117] "RemoveContainer" containerID="e49c0f315fde8fdae62c598637f39581d5b2b44b12219de31ab5be262a233e2b"
Sep 16 12:39:43.995954 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:43.995934    2579 scope.go:117] "RemoveContainer" containerID="e49c0f315fde8fdae62c598637f39581d5b2b44b12219de31ab5be262a233e2b"
Sep 16 12:39:43.996514 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:39:43.996493    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e49c0f315fde8fdae62c598637f39581d5b2b44b12219de31ab5be262a233e2b\": container with ID starting with e49c0f315fde8fdae62c598637f39581d5b2b44b12219de31ab5be262a233e2b not found: ID does not exist" containerID="e49c0f315fde8fdae62c598637f39581d5b2b44b12219de31ab5be262a233e2b"
Sep 16 12:39:43.996593 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:43.996525    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e49c0f315fde8fdae62c598637f39581d5b2b44b12219de31ab5be262a233e2b"} err="failed to get container status \"e49c0f315fde8fdae62c598637f39581d5b2b44b12219de31ab5be262a233e2b\": rpc error: code = NotFound desc = could not find container \"e49c0f315fde8fdae62c598637f39581d5b2b44b12219de31ab5be262a233e2b\": container with ID starting with e49c0f315fde8fdae62c598637f39581d5b2b44b12219de31ab5be262a233e2b not found: ID does not exist"
Sep 16 12:39:44.017563 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:44.017532    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp"]
Sep 16 12:39:44.025217 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:44.025190    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b03a3f-57cdf8454b-mh6gp"]
Sep 16 12:39:44.781905 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:44.781871    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a6b52b6-d72f-4224-81c3-fb5fcb142689" path="/var/lib/kubelet/pods/9a6b52b6-d72f-4224-81c3-fb5fcb142689/volumes"
Sep 16 12:39:47.767040 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:39:47.767005    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:40:07.270487 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:40:07.270445    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:40:19.964277 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:40:19.964233    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:41:10.091208 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:10.091172    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh"]
Sep 16 12:41:10.091783 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:10.091387    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh" podUID="b79691e9-908e-49bb-af57-00b6ca7c17c4" containerName="maas-api" containerID="cri-o://f6c30d135266b63572b5f1ba21ef5b038b9a3f37b3c155e380719f6e1eb1c57d" gracePeriod=30
Sep 16 12:41:10.350487 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:10.350433    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh"
Sep 16 12:41:10.399835 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:10.399805    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhvbq\" (UniqueName: \"kubernetes.io/projected/b79691e9-908e-49bb-af57-00b6ca7c17c4-kube-api-access-fhvbq\") pod \"b79691e9-908e-49bb-af57-00b6ca7c17c4\" (UID: \"b79691e9-908e-49bb-af57-00b6ca7c17c4\") "
Sep 16 12:41:10.399997 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:10.399885    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b79691e9-908e-49bb-af57-00b6ca7c17c4-metrics-certs\") pod \"b79691e9-908e-49bb-af57-00b6ca7c17c4\" (UID: \"b79691e9-908e-49bb-af57-00b6ca7c17c4\") "
Sep 16 12:41:10.399997 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:10.399924    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b79691e9-908e-49bb-af57-00b6ca7c17c4-openshift-service-ca\") pod \"b79691e9-908e-49bb-af57-00b6ca7c17c4\" (UID: \"b79691e9-908e-49bb-af57-00b6ca7c17c4\") "
Sep 16 12:41:10.400108 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:10.400066    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b79691e9-908e-49bb-af57-00b6ca7c17c4-maas-api-tls\") pod \"b79691e9-908e-49bb-af57-00b6ca7c17c4\" (UID: \"b79691e9-908e-49bb-af57-00b6ca7c17c4\") "
Sep 16 12:41:10.400319 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:10.400277    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b79691e9-908e-49bb-af57-00b6ca7c17c4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b79691e9-908e-49bb-af57-00b6ca7c17c4" (UID: "b79691e9-908e-49bb-af57-00b6ca7c17c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:41:10.400475 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:10.400454    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b79691e9-908e-49bb-af57-00b6ca7c17c4-openshift-service-ca\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:41:10.402175 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:10.402083    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b79691e9-908e-49bb-af57-00b6ca7c17c4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b79691e9-908e-49bb-af57-00b6ca7c17c4" (UID: "b79691e9-908e-49bb-af57-00b6ca7c17c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:41:10.402175 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:10.402147    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b79691e9-908e-49bb-af57-00b6ca7c17c4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b79691e9-908e-49bb-af57-00b6ca7c17c4" (UID: "b79691e9-908e-49bb-af57-00b6ca7c17c4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:41:10.402175 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:10.402172    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b79691e9-908e-49bb-af57-00b6ca7c17c4-kube-api-access-fhvbq" (OuterVolumeSpecName: "kube-api-access-fhvbq") pod "b79691e9-908e-49bb-af57-00b6ca7c17c4" (UID: "b79691e9-908e-49bb-af57-00b6ca7c17c4"). InnerVolumeSpecName "kube-api-access-fhvbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:41:10.481755 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:10.481718    2579 generic.go:356] "Generic (PLEG): container finished" podID="b79691e9-908e-49bb-af57-00b6ca7c17c4" containerID="f6c30d135266b63572b5f1ba21ef5b038b9a3f37b3c155e380719f6e1eb1c57d" exitCode=0
Sep 16 12:41:10.481911 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:10.481766    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh"
Sep 16 12:41:10.481911 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:10.481809    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh" event={"ID":"b79691e9-908e-49bb-af57-00b6ca7c17c4","Type":"ContainerDied","Data":"f6c30d135266b63572b5f1ba21ef5b038b9a3f37b3c155e380719f6e1eb1c57d"}
Sep 16 12:41:10.481911 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:10.481867    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh" event={"ID":"b79691e9-908e-49bb-af57-00b6ca7c17c4","Type":"ContainerDied","Data":"f55fc25eefb536d46d7f28529c39b828f5b21b760e466e42f05b8550d2b5bed5"}
Sep 16 12:41:10.481911 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:10.481888    2579 scope.go:117] "RemoveContainer" containerID="f6c30d135266b63572b5f1ba21ef5b038b9a3f37b3c155e380719f6e1eb1c57d"
Sep 16 12:41:10.491948 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:10.491932    2579 scope.go:117] "RemoveContainer" containerID="f6c30d135266b63572b5f1ba21ef5b038b9a3f37b3c155e380719f6e1eb1c57d"
Sep 16 12:41:10.492197 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:41:10.492178    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c30d135266b63572b5f1ba21ef5b038b9a3f37b3c155e380719f6e1eb1c57d\": container with ID starting with f6c30d135266b63572b5f1ba21ef5b038b9a3f37b3c155e380719f6e1eb1c57d not found: ID does not exist" containerID="f6c30d135266b63572b5f1ba21ef5b038b9a3f37b3c155e380719f6e1eb1c57d"
Sep 16 12:41:10.492257 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:10.492207    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c30d135266b63572b5f1ba21ef5b038b9a3f37b3c155e380719f6e1eb1c57d"} err="failed to get container status \"f6c30d135266b63572b5f1ba21ef5b038b9a3f37b3c155e380719f6e1eb1c57d\": rpc error: code = NotFound desc = could not find container \"f6c30d135266b63572b5f1ba21ef5b038b9a3f37b3c155e380719f6e1eb1c57d\": container with ID starting with f6c30d135266b63572b5f1ba21ef5b038b9a3f37b3c155e380719f6e1eb1c57d not found: ID does not exist"
Sep 16 12:41:10.501041 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:10.501011    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhvbq\" (UniqueName: \"kubernetes.io/projected/b79691e9-908e-49bb-af57-00b6ca7c17c4-kube-api-access-fhvbq\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:41:10.501041 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:10.501042    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b79691e9-908e-49bb-af57-00b6ca7c17c4-metrics-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:41:10.501041 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:10.501052    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b79691e9-908e-49bb-af57-00b6ca7c17c4-maas-api-tls\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:41:10.508703 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:10.508675    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh"]
Sep 16 12:41:10.512923 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:10.512897    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-89b7ca-8fc5777c-cj6lh"]
Sep 16 12:41:10.782056 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:10.781964    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b79691e9-908e-49bb-af57-00b6ca7c17c4" path="/var/lib/kubelet/pods/b79691e9-908e-49bb-af57-00b6ca7c17c4/volumes"
Sep 16 12:41:13.462512 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:13.462473    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:41:27.891004 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:27.890972    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"]
Sep 16 12:41:27.891479 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:27.891208    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4" podUID="1ba65312-d842-40fc-be8a-5ef3fb1eca79" containerName="maas-api" containerID="cri-o://a03346a09c7e8b76b6b24af0ce5f6f0d56862aba366487971f9acb596e036e96" gracePeriod=30
Sep 16 12:41:28.130672 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:28.130637    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"
Sep 16 12:41:28.277339 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:28.276489    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbztf\" (UniqueName: \"kubernetes.io/projected/1ba65312-d842-40fc-be8a-5ef3fb1eca79-kube-api-access-tbztf\") pod \"1ba65312-d842-40fc-be8a-5ef3fb1eca79\" (UID: \"1ba65312-d842-40fc-be8a-5ef3fb1eca79\") "
Sep 16 12:41:28.277339 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:28.276569    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ba65312-d842-40fc-be8a-5ef3fb1eca79-openshift-service-ca\") pod \"1ba65312-d842-40fc-be8a-5ef3fb1eca79\" (UID: \"1ba65312-d842-40fc-be8a-5ef3fb1eca79\") "
Sep 16 12:41:28.277339 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:28.276604    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ba65312-d842-40fc-be8a-5ef3fb1eca79-metrics-certs\") pod \"1ba65312-d842-40fc-be8a-5ef3fb1eca79\" (UID: \"1ba65312-d842-40fc-be8a-5ef3fb1eca79\") "
Sep 16 12:41:28.277339 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:28.276690    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ba65312-d842-40fc-be8a-5ef3fb1eca79-maas-api-tls\") pod \"1ba65312-d842-40fc-be8a-5ef3fb1eca79\" (UID: \"1ba65312-d842-40fc-be8a-5ef3fb1eca79\") "
Sep 16 12:41:28.277339 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:28.277306    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba65312-d842-40fc-be8a-5ef3fb1eca79-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1ba65312-d842-40fc-be8a-5ef3fb1eca79" (UID: "1ba65312-d842-40fc-be8a-5ef3fb1eca79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:41:28.279466 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:28.279424    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba65312-d842-40fc-be8a-5ef3fb1eca79-kube-api-access-tbztf" (OuterVolumeSpecName: "kube-api-access-tbztf") pod "1ba65312-d842-40fc-be8a-5ef3fb1eca79" (UID: "1ba65312-d842-40fc-be8a-5ef3fb1eca79"). InnerVolumeSpecName "kube-api-access-tbztf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:41:28.279774 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:28.279735    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ba65312-d842-40fc-be8a-5ef3fb1eca79-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1ba65312-d842-40fc-be8a-5ef3fb1eca79" (UID: "1ba65312-d842-40fc-be8a-5ef3fb1eca79"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:41:28.280079 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:28.280050    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ba65312-d842-40fc-be8a-5ef3fb1eca79-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1ba65312-d842-40fc-be8a-5ef3fb1eca79" (UID: "1ba65312-d842-40fc-be8a-5ef3fb1eca79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:41:28.378414 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:28.378381    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ba65312-d842-40fc-be8a-5ef3fb1eca79-maas-api-tls\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:41:28.378414 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:28.378410    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbztf\" (UniqueName: \"kubernetes.io/projected/1ba65312-d842-40fc-be8a-5ef3fb1eca79-kube-api-access-tbztf\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:41:28.378414 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:28.378420    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ba65312-d842-40fc-be8a-5ef3fb1eca79-openshift-service-ca\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:41:28.378630 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:28.378430    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ba65312-d842-40fc-be8a-5ef3fb1eca79-metrics-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:41:28.586522 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:28.586488    2579 generic.go:356] "Generic (PLEG): container finished" podID="1ba65312-d842-40fc-be8a-5ef3fb1eca79" containerID="a03346a09c7e8b76b6b24af0ce5f6f0d56862aba366487971f9acb596e036e96" exitCode=0
Sep 16 12:41:28.586728 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:28.586532    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"
Sep 16 12:41:28.586728 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:28.586574    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4" event={"ID":"1ba65312-d842-40fc-be8a-5ef3fb1eca79","Type":"ContainerDied","Data":"a03346a09c7e8b76b6b24af0ce5f6f0d56862aba366487971f9acb596e036e96"}
Sep 16 12:41:28.586728 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:28.586623    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4" event={"ID":"1ba65312-d842-40fc-be8a-5ef3fb1eca79","Type":"ContainerDied","Data":"b706b2e22ce95b1634448a2ce8b88ae171af180f0d6b353d8b326c27549d6d11"}
Sep 16 12:41:28.586728 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:28.586646    2579 scope.go:117] "RemoveContainer" containerID="a03346a09c7e8b76b6b24af0ce5f6f0d56862aba366487971f9acb596e036e96"
Sep 16 12:41:28.596394 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:28.596376    2579 scope.go:117] "RemoveContainer" containerID="a03346a09c7e8b76b6b24af0ce5f6f0d56862aba366487971f9acb596e036e96"
Sep 16 12:41:28.596647 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:41:28.596627    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a03346a09c7e8b76b6b24af0ce5f6f0d56862aba366487971f9acb596e036e96\": container with ID starting with a03346a09c7e8b76b6b24af0ce5f6f0d56862aba366487971f9acb596e036e96 not found: ID does not exist" containerID="a03346a09c7e8b76b6b24af0ce5f6f0d56862aba366487971f9acb596e036e96"
Sep 16 12:41:28.596736 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:28.596669    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a03346a09c7e8b76b6b24af0ce5f6f0d56862aba366487971f9acb596e036e96"} err="failed to get container status \"a03346a09c7e8b76b6b24af0ce5f6f0d56862aba366487971f9acb596e036e96\": rpc error: code = NotFound desc = could not find container \"a03346a09c7e8b76b6b24af0ce5f6f0d56862aba366487971f9acb596e036e96\": container with ID starting with a03346a09c7e8b76b6b24af0ce5f6f0d56862aba366487971f9acb596e036e96 not found: ID does not exist"
Sep 16 12:41:28.615260 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:28.615232    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"]
Sep 16 12:41:28.623418 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:28.623395    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-0a9008-bfd9c4bd7-dv6q4"]
Sep 16 12:41:28.781410 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:28.781369    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ba65312-d842-40fc-be8a-5ef3fb1eca79" path="/var/lib/kubelet/pods/1ba65312-d842-40fc-be8a-5ef3fb1eca79/volumes"
Sep 16 12:41:30.769611 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:30.769571    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:41:45.968292 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:45.968258    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq"]
Sep 16 12:41:45.968848 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:45.968511    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq" podUID="cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d" containerName="maas-api" containerID="cri-o://baacecc163667942cfef1c6a348a2a3606daee83d6174739377b81854b7e5574" gracePeriod=30
Sep 16 12:41:46.224166 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:46.224110    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq"
Sep 16 12:41:46.340725 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:46.340689    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-maas-api-tls\") pod \"cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d\" (UID: \"cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d\") "
Sep 16 12:41:46.340879 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:46.340739    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-metrics-certs\") pod \"cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d\" (UID: \"cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d\") "
Sep 16 12:41:46.340879 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:46.340778    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-openshift-service-ca\") pod \"cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d\" (UID: \"cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d\") "
Sep 16 12:41:46.340879 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:46.340806    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4nd7\" (UniqueName: \"kubernetes.io/projected/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-kube-api-access-n4nd7\") pod \"cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d\" (UID: \"cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d\") "
Sep 16 12:41:46.341206 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:46.341181    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d" (UID: "cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:41:46.342937 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:46.342905    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-kube-api-access-n4nd7" (OuterVolumeSpecName: "kube-api-access-n4nd7") pod "cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d" (UID: "cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d"). InnerVolumeSpecName "kube-api-access-n4nd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:41:46.342937 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:46.342920    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d" (UID: "cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:41:46.343120 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:46.342940    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d" (UID: "cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:41:46.442231 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:46.442195    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-openshift-service-ca\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:41:46.442231 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:46.442222    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4nd7\" (UniqueName: \"kubernetes.io/projected/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-kube-api-access-n4nd7\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:41:46.442231 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:46.442233    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-maas-api-tls\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:41:46.442231 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:46.442242    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d-metrics-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:41:46.686836 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:46.686791    2579 generic.go:356] "Generic (PLEG): container finished" podID="cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d" containerID="baacecc163667942cfef1c6a348a2a3606daee83d6174739377b81854b7e5574" exitCode=0
Sep 16 12:41:46.687025 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:46.686918    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq"
Sep 16 12:41:46.687025 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:46.686947    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq" event={"ID":"cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d","Type":"ContainerDied","Data":"baacecc163667942cfef1c6a348a2a3606daee83d6174739377b81854b7e5574"}
Sep 16 12:41:46.687171 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:46.687119    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq" event={"ID":"cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d","Type":"ContainerDied","Data":"16eb846f91019322e28d431d48384f34701f62119eebcfa5f398a8313dd91454"}
Sep 16 12:41:46.687171 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:46.687155    2579 scope.go:117] "RemoveContainer" containerID="baacecc163667942cfef1c6a348a2a3606daee83d6174739377b81854b7e5574"
Sep 16 12:41:46.699559 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:46.699542    2579 scope.go:117] "RemoveContainer" containerID="baacecc163667942cfef1c6a348a2a3606daee83d6174739377b81854b7e5574"
Sep 16 12:41:46.699906 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:41:46.699885    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baacecc163667942cfef1c6a348a2a3606daee83d6174739377b81854b7e5574\": container with ID starting with baacecc163667942cfef1c6a348a2a3606daee83d6174739377b81854b7e5574 not found: ID does not exist" containerID="baacecc163667942cfef1c6a348a2a3606daee83d6174739377b81854b7e5574"
Sep 16 12:41:46.699988 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:46.699919    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baacecc163667942cfef1c6a348a2a3606daee83d6174739377b81854b7e5574"} err="failed to get container status \"baacecc163667942cfef1c6a348a2a3606daee83d6174739377b81854b7e5574\": rpc error: code = NotFound desc = could not find container \"baacecc163667942cfef1c6a348a2a3606daee83d6174739377b81854b7e5574\": container with ID starting with baacecc163667942cfef1c6a348a2a3606daee83d6174739377b81854b7e5574 not found: ID does not exist"
Sep 16 12:41:46.712026 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:46.711996    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq"]
Sep 16 12:41:46.715761 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:46.715739    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0e94e5-5bdb5bd74b-6qzcq"]
Sep 16 12:41:46.782027 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:46.781993    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d" path="/var/lib/kubelet/pods/cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d/volumes"
Sep 16 12:41:48.465873 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:48.465840    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:41:53.002247 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:53.002213    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"]
Sep 16 12:41:53.002732 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:53.002418    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn" podUID="a18e3d54-bb46-4c26-8c7d-b8b4881594b5" containerName="istio-proxy" containerID="cri-o://d6864e4a50ba63f9aa9e68f44c093c340f97f60783d2d93e646ced19091d02ab" gracePeriod=30
Sep 16 12:41:55.148283 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:55.148245    2579 patch_prober.go:28] interesting pod/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 12:41:55.148729 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:55.148309    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn" podUID="a18e3d54-bb46-4c26-8c7d-b8b4881594b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 12:41:58.251449 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.251422    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:41:58.361715 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.361684    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istio-envoy\") pod \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") "
Sep 16 12:41:58.361862 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.361726    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-workload-socket\") pod \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") "
Sep 16 12:41:58.361862 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.361763    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-workload-certs\") pod \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") "
Sep 16 12:41:58.361862 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.361785    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istio-data\") pod \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") "
Sep 16 12:41:58.361862 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.361849    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6mgj\" (UniqueName: \"kubernetes.io/projected/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-kube-api-access-g6mgj\") pod \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") "
Sep 16 12:41:58.362106 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.361879    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istio-podinfo\") pod \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") "
Sep 16 12:41:58.362106 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.361934    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istio-token\") pod \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") "
Sep 16 12:41:58.362106 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.361958    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istiod-ca-cert\") pod \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") "
Sep 16 12:41:58.362259 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.362053    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a18e3d54-bb46-4c26-8c7d-b8b4881594b5" (UID: "a18e3d54-bb46-4c26-8c7d-b8b4881594b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:41:58.362259 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.362123    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a18e3d54-bb46-4c26-8c7d-b8b4881594b5" (UID: "a18e3d54-bb46-4c26-8c7d-b8b4881594b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:41:58.362259 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.362140    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istio-data" (OuterVolumeSpecName: "istio-data") pod "a18e3d54-bb46-4c26-8c7d-b8b4881594b5" (UID: "a18e3d54-bb46-4c26-8c7d-b8b4881594b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:41:58.362259 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.362163    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-credential-socket\") pod \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\" (UID: \"a18e3d54-bb46-4c26-8c7d-b8b4881594b5\") "
Sep 16 12:41:58.362551 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.362494    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a18e3d54-bb46-4c26-8c7d-b8b4881594b5" (UID: "a18e3d54-bb46-4c26-8c7d-b8b4881594b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:41:58.362551 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.362505    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-workload-socket\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:41:58.362551 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.362545    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-workload-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:41:58.362808 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.362564    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istio-data\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:41:58.362808 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.362543    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a18e3d54-bb46-4c26-8c7d-b8b4881594b5" (UID: "a18e3d54-bb46-4c26-8c7d-b8b4881594b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:41:58.363938 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.363917    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a18e3d54-bb46-4c26-8c7d-b8b4881594b5" (UID: "a18e3d54-bb46-4c26-8c7d-b8b4881594b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 12:41:58.364160 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.364145    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a18e3d54-bb46-4c26-8c7d-b8b4881594b5" (UID: "a18e3d54-bb46-4c26-8c7d-b8b4881594b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 12:41:58.364446 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.364424    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-kube-api-access-g6mgj" (OuterVolumeSpecName: "kube-api-access-g6mgj") pod "a18e3d54-bb46-4c26-8c7d-b8b4881594b5" (UID: "a18e3d54-bb46-4c26-8c7d-b8b4881594b5"). InnerVolumeSpecName "kube-api-access-g6mgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:41:58.364514 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.364423    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istio-token" (OuterVolumeSpecName: "istio-token") pod "a18e3d54-bb46-4c26-8c7d-b8b4881594b5" (UID: "a18e3d54-bb46-4c26-8c7d-b8b4881594b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:41:58.463176 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.463141    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istio-envoy\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:41:58.463176 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.463172    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6mgj\" (UniqueName: \"kubernetes.io/projected/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-kube-api-access-g6mgj\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:41:58.463176 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.463184    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istio-podinfo\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:41:58.463401 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.463192    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istio-token\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:41:58.463401 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.463200    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-istiod-ca-cert\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:41:58.463401 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.463208    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a18e3d54-bb46-4c26-8c7d-b8b4881594b5-credential-socket\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:41:58.757633 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.757551    2579 generic.go:356] "Generic (PLEG): container finished" podID="a18e3d54-bb46-4c26-8c7d-b8b4881594b5" containerID="d6864e4a50ba63f9aa9e68f44c093c340f97f60783d2d93e646ced19091d02ab" exitCode=0
Sep 16 12:41:58.757633 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.757604    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"
Sep 16 12:41:58.757844 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.757631    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn" event={"ID":"a18e3d54-bb46-4c26-8c7d-b8b4881594b5","Type":"ContainerDied","Data":"d6864e4a50ba63f9aa9e68f44c093c340f97f60783d2d93e646ced19091d02ab"}
Sep 16 12:41:58.757844 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.757709    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn" event={"ID":"a18e3d54-bb46-4c26-8c7d-b8b4881594b5","Type":"ContainerDied","Data":"735880ea7ab2ab2f0d350b75638d7e178a7f81ec000029529ff0ebad7837aaf5"}
Sep 16 12:41:58.757844 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.757732    2579 scope.go:117] "RemoveContainer" containerID="d6864e4a50ba63f9aa9e68f44c093c340f97f60783d2d93e646ced19091d02ab"
Sep 16 12:41:58.769702 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.769685    2579 scope.go:117] "RemoveContainer" containerID="79d2e3fda5fd9c1fc2f4feacbbef2ed13b4dc123ab6868626da5d3b2004d0bf6"
Sep 16 12:41:58.778678 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.778584    2579 scope.go:117] "RemoveContainer" containerID="d6864e4a50ba63f9aa9e68f44c093c340f97f60783d2d93e646ced19091d02ab"
Sep 16 12:41:58.779059 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:41:58.779030    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6864e4a50ba63f9aa9e68f44c093c340f97f60783d2d93e646ced19091d02ab\": container with ID starting with d6864e4a50ba63f9aa9e68f44c093c340f97f60783d2d93e646ced19091d02ab not found: ID does not exist" containerID="d6864e4a50ba63f9aa9e68f44c093c340f97f60783d2d93e646ced19091d02ab"
Sep 16 12:41:58.779223 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.779200    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6864e4a50ba63f9aa9e68f44c093c340f97f60783d2d93e646ced19091d02ab"} err="failed to get container status \"d6864e4a50ba63f9aa9e68f44c093c340f97f60783d2d93e646ced19091d02ab\": rpc error: code = NotFound desc = could not find container \"d6864e4a50ba63f9aa9e68f44c093c340f97f60783d2d93e646ced19091d02ab\": container with ID starting with d6864e4a50ba63f9aa9e68f44c093c340f97f60783d2d93e646ced19091d02ab not found: ID does not exist"
Sep 16 12:41:58.779338 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.779317    2579 scope.go:117] "RemoveContainer" containerID="79d2e3fda5fd9c1fc2f4feacbbef2ed13b4dc123ab6868626da5d3b2004d0bf6"
Sep 16 12:41:58.779685 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:41:58.779638    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79d2e3fda5fd9c1fc2f4feacbbef2ed13b4dc123ab6868626da5d3b2004d0bf6\": container with ID starting with 79d2e3fda5fd9c1fc2f4feacbbef2ed13b4dc123ab6868626da5d3b2004d0bf6 not found: ID does not exist" containerID="79d2e3fda5fd9c1fc2f4feacbbef2ed13b4dc123ab6868626da5d3b2004d0bf6"
Sep 16 12:41:58.779754 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.779694    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d2e3fda5fd9c1fc2f4feacbbef2ed13b4dc123ab6868626da5d3b2004d0bf6"} err="failed to get container status \"79d2e3fda5fd9c1fc2f4feacbbef2ed13b4dc123ab6868626da5d3b2004d0bf6\": rpc error: code = NotFound desc = could not find container \"79d2e3fda5fd9c1fc2f4feacbbef2ed13b4dc123ab6868626da5d3b2004d0bf6\": container with ID starting with 79d2e3fda5fd9c1fc2f4feacbbef2ed13b4dc123ab6868626da5d3b2004d0bf6 not found: ID does not exist"
Sep 16 12:41:58.783094 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.783072    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"]
Sep 16 12:41:58.785531 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.785513    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-0e94e5-openshift-default-5c6d64c646-fx6hn"]
Sep 16 12:41:58.962180 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:41:58.962143    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:42:00.780319 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:42:00.780287    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a18e3d54-bb46-4c26-8c7d-b8b4881594b5" path="/var/lib/kubelet/pods/a18e3d54-bb46-4c26-8c7d-b8b4881594b5/volumes"
Sep 16 12:43:23.068829 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:43:23.068796    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:43:49.263391 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:43:49.263308    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:43:56.868385 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:43:56.868345    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:44:05.063885 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:44:05.063853    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:44:16.173580 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:44:16.173539    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:44:22.460228 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:44:22.460193    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:44:48.459238 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:44:48.459197    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:44:54.163010 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:44:54.162964    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:46:19.612578 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:46:19.612539    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:46:27.713480 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:46:27.713442    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:46:40.809377 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:46:40.809340    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:46:50.410692 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:46:50.410581    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:47:29.439952 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:29.439913    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:47:38.417859 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:38.417826    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:47:47.809693 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:47.809645    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:47:48.104414 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.104375    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-75d578b874-5wfgj"]
Sep 16 12:47:48.104767 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.104753    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b79691e9-908e-49bb-af57-00b6ca7c17c4" containerName="maas-api"
Sep 16 12:47:48.104818 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.104769    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b79691e9-908e-49bb-af57-00b6ca7c17c4" containerName="maas-api"
Sep 16 12:47:48.104818 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.104788    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ba65312-d842-40fc-be8a-5ef3fb1eca79" containerName="maas-api"
Sep 16 12:47:48.104818 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.104794    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ba65312-d842-40fc-be8a-5ef3fb1eca79" containerName="maas-api"
Sep 16 12:47:48.104818 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.104800    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a18e3d54-bb46-4c26-8c7d-b8b4881594b5" containerName="istio-proxy"
Sep 16 12:47:48.104818 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.104805    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a18e3d54-bb46-4c26-8c7d-b8b4881594b5" containerName="istio-proxy"
Sep 16 12:47:48.104818 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.104815    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d" containerName="maas-api"
Sep 16 12:47:48.104818 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.104820    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d" containerName="maas-api"
Sep 16 12:47:48.105076 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.104826    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a6b52b6-d72f-4224-81c3-fb5fcb142689" containerName="maas-api"
Sep 16 12:47:48.105076 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.104831    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a6b52b6-d72f-4224-81c3-fb5fcb142689" containerName="maas-api"
Sep 16 12:47:48.105076 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.104844    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a18e3d54-bb46-4c26-8c7d-b8b4881594b5" containerName="istio-proxy"
Sep 16 12:47:48.105076 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.104849    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a18e3d54-bb46-4c26-8c7d-b8b4881594b5" containerName="istio-proxy"
Sep 16 12:47:48.105076 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.104907    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b79691e9-908e-49bb-af57-00b6ca7c17c4" containerName="maas-api"
Sep 16 12:47:48.105076 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.104914    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a18e3d54-bb46-4c26-8c7d-b8b4881594b5" containerName="istio-proxy"
Sep 16 12:47:48.105076 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.104922    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cde23e4a-dbc2-4b0c-90aa-4aa34ba14c8d" containerName="maas-api"
Sep 16 12:47:48.105076 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.104928    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ba65312-d842-40fc-be8a-5ef3fb1eca79" containerName="maas-api"
Sep 16 12:47:48.105076 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.104934    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a6b52b6-d72f-4224-81c3-fb5fcb142689" containerName="maas-api"
Sep 16 12:47:48.107886 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.107870    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75d578b874-5wfgj"
Sep 16 12:47:48.111892 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.111870    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 12:47:48.111892 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.111892    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 12:47:48.112135 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.111905    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4gkmq\""
Sep 16 12:47:48.112135 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.111915    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 16 12:47:48.112135 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.111961    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 12:47:48.128922 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.128890    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75d578b874-5wfgj"]
Sep 16 12:47:48.131563 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.131535    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5bfbbdfcb6-4szq2"]
Sep 16 12:47:48.132107 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.132086    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a18e3d54-bb46-4c26-8c7d-b8b4881594b5" containerName="istio-proxy"
Sep 16 12:47:48.135429 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.135327    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-4szq2"
Sep 16 12:47:48.139813 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.139784    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bfbbdfcb6-4szq2"]
Sep 16 12:47:48.179054 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.179027    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d4dcbf2-124c-4684-884e-e6127ac702ce-metrics-certs\") pod \"maas-api-75d578b874-5wfgj\" (UID: \"0d4dcbf2-124c-4684-884e-e6127ac702ce\") " pod="odh-ai-gateway-infra/maas-api-75d578b874-5wfgj"
Sep 16 12:47:48.179195 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.179066    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d4dcbf2-124c-4684-884e-e6127ac702ce-openshift-service-ca\") pod \"maas-api-75d578b874-5wfgj\" (UID: \"0d4dcbf2-124c-4684-884e-e6127ac702ce\") " pod="odh-ai-gateway-infra/maas-api-75d578b874-5wfgj"
Sep 16 12:47:48.179195 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.179122    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d4dcbf2-124c-4684-884e-e6127ac702ce-maas-api-tls\") pod \"maas-api-75d578b874-5wfgj\" (UID: \"0d4dcbf2-124c-4684-884e-e6127ac702ce\") " pod="odh-ai-gateway-infra/maas-api-75d578b874-5wfgj"
Sep 16 12:47:48.179195 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.179146    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbz5k\" (UniqueName: \"kubernetes.io/projected/0d4dcbf2-124c-4684-884e-e6127ac702ce-kube-api-access-hbz5k\") pod \"maas-api-75d578b874-5wfgj\" (UID: \"0d4dcbf2-124c-4684-884e-e6127ac702ce\") " pod="odh-ai-gateway-infra/maas-api-75d578b874-5wfgj"
Sep 16 12:47:48.280546 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.280513    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d4dcbf2-124c-4684-884e-e6127ac702ce-metrics-certs\") pod \"maas-api-75d578b874-5wfgj\" (UID: \"0d4dcbf2-124c-4684-884e-e6127ac702ce\") " pod="odh-ai-gateway-infra/maas-api-75d578b874-5wfgj"
Sep 16 12:47:48.280546 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.280549    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx9x9\" (UniqueName: \"kubernetes.io/projected/2c290968-4f5d-496c-9c7e-912c9d91ae17-kube-api-access-tx9x9\") pod \"maas-api-5bfbbdfcb6-4szq2\" (UID: \"2c290968-4f5d-496c-9c7e-912c9d91ae17\") " pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-4szq2"
Sep 16 12:47:48.280786 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.280579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d4dcbf2-124c-4684-884e-e6127ac702ce-openshift-service-ca\") pod \"maas-api-75d578b874-5wfgj\" (UID: \"0d4dcbf2-124c-4684-884e-e6127ac702ce\") " pod="odh-ai-gateway-infra/maas-api-75d578b874-5wfgj"
Sep 16 12:47:48.280786 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.280623    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d4dcbf2-124c-4684-884e-e6127ac702ce-maas-api-tls\") pod \"maas-api-75d578b874-5wfgj\" (UID: \"0d4dcbf2-124c-4684-884e-e6127ac702ce\") " pod="odh-ai-gateway-infra/maas-api-75d578b874-5wfgj"
Sep 16 12:47:48.280786 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.280644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbz5k\" (UniqueName: \"kubernetes.io/projected/0d4dcbf2-124c-4684-884e-e6127ac702ce-kube-api-access-hbz5k\") pod \"maas-api-75d578b874-5wfgj\" (UID: \"0d4dcbf2-124c-4684-884e-e6127ac702ce\") " pod="odh-ai-gateway-infra/maas-api-75d578b874-5wfgj"
Sep 16 12:47:48.280786 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.280703    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c290968-4f5d-496c-9c7e-912c9d91ae17-metrics-certs\") pod \"maas-api-5bfbbdfcb6-4szq2\" (UID: \"2c290968-4f5d-496c-9c7e-912c9d91ae17\") " pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-4szq2"
Sep 16 12:47:48.280786 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.280730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c290968-4f5d-496c-9c7e-912c9d91ae17-maas-api-tls\") pod \"maas-api-5bfbbdfcb6-4szq2\" (UID: \"2c290968-4f5d-496c-9c7e-912c9d91ae17\") " pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-4szq2"
Sep 16 12:47:48.280786 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.280754    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c290968-4f5d-496c-9c7e-912c9d91ae17-openshift-service-ca\") pod \"maas-api-5bfbbdfcb6-4szq2\" (UID: \"2c290968-4f5d-496c-9c7e-912c9d91ae17\") " pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-4szq2"
Sep 16 12:47:48.281258 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.281234    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d4dcbf2-124c-4684-884e-e6127ac702ce-openshift-service-ca\") pod \"maas-api-75d578b874-5wfgj\" (UID: \"0d4dcbf2-124c-4684-884e-e6127ac702ce\") " pod="odh-ai-gateway-infra/maas-api-75d578b874-5wfgj"
Sep 16 12:47:48.283000 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.282976    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d4dcbf2-124c-4684-884e-e6127ac702ce-maas-api-tls\") pod \"maas-api-75d578b874-5wfgj\" (UID: \"0d4dcbf2-124c-4684-884e-e6127ac702ce\") " pod="odh-ai-gateway-infra/maas-api-75d578b874-5wfgj"
Sep 16 12:47:48.283089 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.283036    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d4dcbf2-124c-4684-884e-e6127ac702ce-metrics-certs\") pod \"maas-api-75d578b874-5wfgj\" (UID: \"0d4dcbf2-124c-4684-884e-e6127ac702ce\") " pod="odh-ai-gateway-infra/maas-api-75d578b874-5wfgj"
Sep 16 12:47:48.288828 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.288804    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbz5k\" (UniqueName: \"kubernetes.io/projected/0d4dcbf2-124c-4684-884e-e6127ac702ce-kube-api-access-hbz5k\") pod \"maas-api-75d578b874-5wfgj\" (UID: \"0d4dcbf2-124c-4684-884e-e6127ac702ce\") " pod="odh-ai-gateway-infra/maas-api-75d578b874-5wfgj"
Sep 16 12:47:48.381867 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.381769    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c290968-4f5d-496c-9c7e-912c9d91ae17-metrics-certs\") pod \"maas-api-5bfbbdfcb6-4szq2\" (UID: \"2c290968-4f5d-496c-9c7e-912c9d91ae17\") " pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-4szq2"
Sep 16 12:47:48.381867 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.381822    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c290968-4f5d-496c-9c7e-912c9d91ae17-maas-api-tls\") pod \"maas-api-5bfbbdfcb6-4szq2\" (UID: \"2c290968-4f5d-496c-9c7e-912c9d91ae17\") " pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-4szq2"
Sep 16 12:47:48.381867 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.381856    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c290968-4f5d-496c-9c7e-912c9d91ae17-openshift-service-ca\") pod \"maas-api-5bfbbdfcb6-4szq2\" (UID: \"2c290968-4f5d-496c-9c7e-912c9d91ae17\") " pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-4szq2"
Sep 16 12:47:48.382152 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.381900    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx9x9\" (UniqueName: \"kubernetes.io/projected/2c290968-4f5d-496c-9c7e-912c9d91ae17-kube-api-access-tx9x9\") pod \"maas-api-5bfbbdfcb6-4szq2\" (UID: \"2c290968-4f5d-496c-9c7e-912c9d91ae17\") " pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-4szq2"
Sep 16 12:47:48.382576 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.382550    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c290968-4f5d-496c-9c7e-912c9d91ae17-openshift-service-ca\") pod \"maas-api-5bfbbdfcb6-4szq2\" (UID: \"2c290968-4f5d-496c-9c7e-912c9d91ae17\") " pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-4szq2"
Sep 16 12:47:48.384206 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.384183    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c290968-4f5d-496c-9c7e-912c9d91ae17-metrics-certs\") pod \"maas-api-5bfbbdfcb6-4szq2\" (UID: \"2c290968-4f5d-496c-9c7e-912c9d91ae17\") " pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-4szq2"
Sep 16 12:47:48.384349 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.384327    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c290968-4f5d-496c-9c7e-912c9d91ae17-maas-api-tls\") pod \"maas-api-5bfbbdfcb6-4szq2\" (UID: \"2c290968-4f5d-496c-9c7e-912c9d91ae17\") " pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-4szq2"
Sep 16 12:47:48.393592 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.393570    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx9x9\" (UniqueName: \"kubernetes.io/projected/2c290968-4f5d-496c-9c7e-912c9d91ae17-kube-api-access-tx9x9\") pod \"maas-api-5bfbbdfcb6-4szq2\" (UID: \"2c290968-4f5d-496c-9c7e-912c9d91ae17\") " pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-4szq2"
Sep 16 12:47:48.421313 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.421284    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75d578b874-5wfgj"
Sep 16 12:47:48.446457 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.446426    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-4szq2"
Sep 16 12:47:48.706684 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.702174    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-75d578b874-5wfgj"]
Sep 16 12:47:48.708786 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.708503    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 12:47:48.732337 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:48.732308    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5bfbbdfcb6-4szq2"]
Sep 16 12:47:48.733607 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:47:48.733566    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c290968_4f5d_496c_9c7e_912c9d91ae17.slice/crio-f5b2e8690aba7f7fbbad1dc8364e3c9e231818611ff0f4c611a3e50e56aa621a WatchSource:0}: Error finding container f5b2e8690aba7f7fbbad1dc8364e3c9e231818611ff0f4c611a3e50e56aa621a: Status 404 returned error can't find the container with id f5b2e8690aba7f7fbbad1dc8364e3c9e231818611ff0f4c611a3e50e56aa621a
Sep 16 12:47:49.668727 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:49.668690    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-4szq2" event={"ID":"2c290968-4f5d-496c-9c7e-912c9d91ae17","Type":"ContainerStarted","Data":"7ce7e157199c1649eff8693219de7dfd7c5ca38df32068054022414553d21e93"}
Sep 16 12:47:49.669188 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:49.668747    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-4szq2" event={"ID":"2c290968-4f5d-496c-9c7e-912c9d91ae17","Type":"ContainerStarted","Data":"f5b2e8690aba7f7fbbad1dc8364e3c9e231818611ff0f4c611a3e50e56aa621a"}
Sep 16 12:47:49.669188 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:49.668848    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-4szq2"
Sep 16 12:47:49.670260 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:49.670241    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75d578b874-5wfgj" event={"ID":"0d4dcbf2-124c-4684-884e-e6127ac702ce","Type":"ContainerStarted","Data":"40314604249dcb5c840f336a1731e5093bb96049cd77ce614d6aea18f32fbbc7"}
Sep 16 12:47:49.670332 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:49.670277    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75d578b874-5wfgj" event={"ID":"0d4dcbf2-124c-4684-884e-e6127ac702ce","Type":"ContainerStarted","Data":"83e612f46299d17e2e1c50ea76dbac125544cc5a110d4ae0b7c61e1e17222090"}
Sep 16 12:47:49.670332 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:49.670298    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-75d578b874-5wfgj"
Sep 16 12:47:49.687606 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:49.687565    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-4szq2" podStartSLOduration=1.188922606 podStartE2EDuration="1.687552088s" totalImagesPullingTime="498.629482ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:47:49.686862211 +0000 UTC m=+2663.387933538" watchObservedRunningTime="2026-09-16 12:47:49.687552088 +0000 UTC m=+2663.388623413"
Sep 16 12:47:49.711806 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:49.711758    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-75d578b874-5wfgj" podStartSLOduration=1.199105991 podStartE2EDuration="1.711742418s" totalImagesPullingTime="512.636427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 12:47:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:47:49.710791645 +0000 UTC m=+2663.411862965" watchObservedRunningTime="2026-09-16 12:47:49.711742418 +0000 UTC m=+2663.412813757"
Sep 16 12:47:55.609151 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:55.609109    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:47:55.681702 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:55.681647    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-75d578b874-5wfgj"
Sep 16 12:47:55.681863 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:55.681849    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5bfbbdfcb6-4szq2"
Sep 16 12:47:55.748025 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:55.747987    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-75d578b874-5wfgj"]
Sep 16 12:47:55.748195 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:55.748166    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-75d578b874-5wfgj" podUID="0d4dcbf2-124c-4684-884e-e6127ac702ce" containerName="maas-api" containerID="cri-o://40314604249dcb5c840f336a1731e5093bb96049cd77ce614d6aea18f32fbbc7" gracePeriod=30
Sep 16 12:47:55.993042 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:55.993015    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75d578b874-5wfgj"
Sep 16 12:47:56.154041 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:56.153943    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d4dcbf2-124c-4684-884e-e6127ac702ce-maas-api-tls\") pod \"0d4dcbf2-124c-4684-884e-e6127ac702ce\" (UID: \"0d4dcbf2-124c-4684-884e-e6127ac702ce\") "
Sep 16 12:47:56.154041 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:56.154019    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d4dcbf2-124c-4684-884e-e6127ac702ce-openshift-service-ca\") pod \"0d4dcbf2-124c-4684-884e-e6127ac702ce\" (UID: \"0d4dcbf2-124c-4684-884e-e6127ac702ce\") "
Sep 16 12:47:56.154275 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:56.154070    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d4dcbf2-124c-4684-884e-e6127ac702ce-metrics-certs\") pod \"0d4dcbf2-124c-4684-884e-e6127ac702ce\" (UID: \"0d4dcbf2-124c-4684-884e-e6127ac702ce\") "
Sep 16 12:47:56.154275 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:56.154104    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbz5k\" (UniqueName: \"kubernetes.io/projected/0d4dcbf2-124c-4684-884e-e6127ac702ce-kube-api-access-hbz5k\") pod \"0d4dcbf2-124c-4684-884e-e6127ac702ce\" (UID: \"0d4dcbf2-124c-4684-884e-e6127ac702ce\") "
Sep 16 12:47:56.154432 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:56.154409    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d4dcbf2-124c-4684-884e-e6127ac702ce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0d4dcbf2-124c-4684-884e-e6127ac702ce" (UID: "0d4dcbf2-124c-4684-884e-e6127ac702ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 12:47:56.156064 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:56.156039    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d4dcbf2-124c-4684-884e-e6127ac702ce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0d4dcbf2-124c-4684-884e-e6127ac702ce" (UID: "0d4dcbf2-124c-4684-884e-e6127ac702ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:47:56.156179 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:56.156079    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d4dcbf2-124c-4684-884e-e6127ac702ce-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0d4dcbf2-124c-4684-884e-e6127ac702ce" (UID: "0d4dcbf2-124c-4684-884e-e6127ac702ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 12:47:56.156179 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:56.156082    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d4dcbf2-124c-4684-884e-e6127ac702ce-kube-api-access-hbz5k" (OuterVolumeSpecName: "kube-api-access-hbz5k") pod "0d4dcbf2-124c-4684-884e-e6127ac702ce" (UID: "0d4dcbf2-124c-4684-884e-e6127ac702ce"). InnerVolumeSpecName "kube-api-access-hbz5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 12:47:56.255006 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:56.254973    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d4dcbf2-124c-4684-884e-e6127ac702ce-maas-api-tls\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:47:56.255006 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:56.255001    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d4dcbf2-124c-4684-884e-e6127ac702ce-openshift-service-ca\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:47:56.255006 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:56.255011    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d4dcbf2-124c-4684-884e-e6127ac702ce-metrics-certs\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:47:56.255235 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:56.255021    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbz5k\" (UniqueName: \"kubernetes.io/projected/0d4dcbf2-124c-4684-884e-e6127ac702ce-kube-api-access-hbz5k\") on node \"ip-10-0-137-30.ec2.internal\" DevicePath \"\""
Sep 16 12:47:56.711708 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:56.711650    2579 generic.go:356] "Generic (PLEG): container finished" podID="0d4dcbf2-124c-4684-884e-e6127ac702ce" containerID="40314604249dcb5c840f336a1731e5093bb96049cd77ce614d6aea18f32fbbc7" exitCode=0
Sep 16 12:47:56.711708 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:56.711711    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-75d578b874-5wfgj"
Sep 16 12:47:56.712278 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:56.711775    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75d578b874-5wfgj" event={"ID":"0d4dcbf2-124c-4684-884e-e6127ac702ce","Type":"ContainerDied","Data":"40314604249dcb5c840f336a1731e5093bb96049cd77ce614d6aea18f32fbbc7"}
Sep 16 12:47:56.712278 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:56.711843    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-75d578b874-5wfgj" event={"ID":"0d4dcbf2-124c-4684-884e-e6127ac702ce","Type":"ContainerDied","Data":"83e612f46299d17e2e1c50ea76dbac125544cc5a110d4ae0b7c61e1e17222090"}
Sep 16 12:47:56.712278 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:56.711878    2579 scope.go:117] "RemoveContainer" containerID="40314604249dcb5c840f336a1731e5093bb96049cd77ce614d6aea18f32fbbc7"
Sep 16 12:47:56.725465 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:56.725392    2579 scope.go:117] "RemoveContainer" containerID="40314604249dcb5c840f336a1731e5093bb96049cd77ce614d6aea18f32fbbc7"
Sep 16 12:47:56.725801 ip-10-0-137-30 kubenswrapper[2579]: E0916 12:47:56.725779    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40314604249dcb5c840f336a1731e5093bb96049cd77ce614d6aea18f32fbbc7\": container with ID starting with 40314604249dcb5c840f336a1731e5093bb96049cd77ce614d6aea18f32fbbc7 not found: ID does not exist" containerID="40314604249dcb5c840f336a1731e5093bb96049cd77ce614d6aea18f32fbbc7"
Sep 16 12:47:56.725898 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:56.725811    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40314604249dcb5c840f336a1731e5093bb96049cd77ce614d6aea18f32fbbc7"} err="failed to get container status \"40314604249dcb5c840f336a1731e5093bb96049cd77ce614d6aea18f32fbbc7\": rpc error: code = NotFound desc = could not find container \"40314604249dcb5c840f336a1731e5093bb96049cd77ce614d6aea18f32fbbc7\": container with ID starting with 40314604249dcb5c840f336a1731e5093bb96049cd77ce614d6aea18f32fbbc7 not found: ID does not exist"
Sep 16 12:47:56.737094 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:56.737066    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-75d578b874-5wfgj"]
Sep 16 12:47:56.740983 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:56.740953    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-75d578b874-5wfgj"]
Sep 16 12:47:56.783722 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:47:56.783687    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d4dcbf2-124c-4684-884e-e6127ac702ce" path="/var/lib/kubelet/pods/0d4dcbf2-124c-4684-884e-e6127ac702ce/volumes"
Sep 16 12:48:02.801154 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:02.801116    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:48:11.106976 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:11.106937    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:48:15.301936 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:15.301899    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-flrsc"]
Sep 16 12:48:24.496739 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:24.496643    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-n5f8s_f7b110d0-4396-476b-a69b-91f00e2405cf/manager/0.log"
Sep 16 12:48:24.617417 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:24.617387    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-gwhkn_2b325748-891a-4d74-9bd1-ccbc177d05f2/manager/0.log"
Sep 16 12:48:24.844698 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:24.844646    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-njsx5_6aaabad9-9969-41bf-a8e3-620e6f1772f5/server/0.log"
Sep 16 12:48:25.069610 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:25.069583    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b5b57bf8b-2446x_c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4/manager/0.log"
Sep 16 12:48:25.075853 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:25.075834    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b5b57bf8b-2446x_c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4/copy-manifests/0.log"
Sep 16 12:48:25.680580 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:25.680541    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5bfbbdfcb6-4szq2_2c290968-4f5d-496c-9c7e-912c9d91ae17/maas-api/0.log"
Sep 16 12:48:27.240479 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:27.240449    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-84gmp_18161926-6fd0-4968-a37a-e34fe2d290ed/kuadrant-console-plugin/0.log"
Sep 16 12:48:27.494165 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:27.494089    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-flrsc_e5be99d1-3eed-4b4f-96d4-813e2b8a6775/limitador/0.log"
Sep 16 12:48:27.988963 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:27.988935    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp_54b8d903-4ad2-41f5-8db2-a22e9c8e76ca/istio-proxy/0.log"
Sep 16 12:48:28.440048 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:28.440020    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h6qjt_6ab958a3-431d-4406-953e-c5440bb50bb5/istio-proxy/1.log"
Sep 16 12:48:29.361724 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:29.361692    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-fscjx_25e0b03f-c77f-4a62-9eff-0a4c11331d8f/main/0.log"
Sep 16 12:48:38.606381 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:38.606351    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qgx6n_fbbe06db-6359-4c0d-aa65-d07fa8245950/global-pull-secret-syncer/0.log"
Sep 16 12:48:38.720739 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:38.720709    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-htk8n_14653c0f-3cc1-4527-b610-bb5c65e4f7db/konnectivity-agent/0.log"
Sep 16 12:48:38.845362 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:38.845333    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-30.ec2.internal_3d94b98261326c611e79bccb709825c5/haproxy/0.log"
Sep 16 12:48:43.340519 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:43.340491    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-84gmp_18161926-6fd0-4968-a37a-e34fe2d290ed/kuadrant-console-plugin/0.log"
Sep 16 12:48:43.413208 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:43.413178    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-flrsc_e5be99d1-3eed-4b4f-96d4-813e2b8a6775/limitador/0.log"
Sep 16 12:48:44.976804 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:44.976772    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fbf3df07-a75e-413d-8f60-ed4bd2bb9e08/alertmanager/0.log"
Sep 16 12:48:45.027160 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:45.027135    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fbf3df07-a75e-413d-8f60-ed4bd2bb9e08/config-reloader/0.log"
Sep 16 12:48:45.064475 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:45.064453    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fbf3df07-a75e-413d-8f60-ed4bd2bb9e08/kube-rbac-proxy-web/0.log"
Sep 16 12:48:45.091973 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:45.091944    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fbf3df07-a75e-413d-8f60-ed4bd2bb9e08/kube-rbac-proxy/0.log"
Sep 16 12:48:45.121825 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:45.121798    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fbf3df07-a75e-413d-8f60-ed4bd2bb9e08/kube-rbac-proxy-metric/0.log"
Sep 16 12:48:45.157718 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:45.157695    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fbf3df07-a75e-413d-8f60-ed4bd2bb9e08/prom-label-proxy/0.log"
Sep 16 12:48:45.180541 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:45.180515    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fbf3df07-a75e-413d-8f60-ed4bd2bb9e08/init-config-reloader/0.log"
Sep 16 12:48:45.336981 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:45.336953    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-dc766c59d-hfznn_02da3487-ee2b-4d3d-9b0a-8414a6c0c653/metrics-server/0.log"
Sep 16 12:48:45.406298 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:45.406266    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n2r6c_8ae8d233-edab-4270-b326-6fc1277bfb20/node-exporter/0.log"
Sep 16 12:48:45.432233 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:45.432198    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n2r6c_8ae8d233-edab-4270-b326-6fc1277bfb20/kube-rbac-proxy/0.log"
Sep 16 12:48:45.457948 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:45.457924    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n2r6c_8ae8d233-edab-4270-b326-6fc1277bfb20/init-textfile/0.log"
Sep 16 12:48:46.207892 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:46.207857    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-xctwt_40c34b1f-ed4a-4671-92c2-b27588a691c0/prometheus-operator-admission-webhook/0.log"
Sep 16 12:48:48.744183 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:48.744063    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2psvv/perf-node-gather-daemonset-kpc8h"]
Sep 16 12:48:48.745072 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:48.745044    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d4dcbf2-124c-4684-884e-e6127ac702ce" containerName="maas-api"
Sep 16 12:48:48.745072 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:48.745074    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d4dcbf2-124c-4684-884e-e6127ac702ce" containerName="maas-api"
Sep 16 12:48:48.745322 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:48.745304    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d4dcbf2-124c-4684-884e-e6127ac702ce" containerName="maas-api"
Sep 16 12:48:48.749359 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:48.749332    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-kpc8h"
Sep 16 12:48:48.752459 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:48.752437    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2psvv\"/\"default-dockercfg-nq47s\""
Sep 16 12:48:48.752875 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:48.752853    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2psvv\"/\"kube-root-ca.crt\""
Sep 16 12:48:48.752977 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:48.752853    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2psvv\"/\"openshift-service-ca.crt\""
Sep 16 12:48:48.762050 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:48.762026    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2psvv/perf-node-gather-daemonset-kpc8h"]
Sep 16 12:48:48.815980 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:48.815943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fe57897f-ea22-4f3e-a497-19c0649debc5-podres\") pod \"perf-node-gather-daemonset-kpc8h\" (UID: \"fe57897f-ea22-4f3e-a497-19c0649debc5\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-kpc8h"
Sep 16 12:48:48.816159 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:48.815985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe57897f-ea22-4f3e-a497-19c0649debc5-lib-modules\") pod \"perf-node-gather-daemonset-kpc8h\" (UID: \"fe57897f-ea22-4f3e-a497-19c0649debc5\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-kpc8h"
Sep 16 12:48:48.816159 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:48.816044    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fe57897f-ea22-4f3e-a497-19c0649debc5-proc\") pod \"perf-node-gather-daemonset-kpc8h\" (UID: \"fe57897f-ea22-4f3e-a497-19c0649debc5\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-kpc8h"
Sep 16 12:48:48.816159 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:48.816134    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btdnp\" (UniqueName: \"kubernetes.io/projected/fe57897f-ea22-4f3e-a497-19c0649debc5-kube-api-access-btdnp\") pod \"perf-node-gather-daemonset-kpc8h\" (UID: \"fe57897f-ea22-4f3e-a497-19c0649debc5\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-kpc8h"
Sep 16 12:48:48.816273 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:48.816192    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe57897f-ea22-4f3e-a497-19c0649debc5-sys\") pod \"perf-node-gather-daemonset-kpc8h\" (UID: \"fe57897f-ea22-4f3e-a497-19c0649debc5\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-kpc8h"
Sep 16 12:48:48.917385 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:48.917343    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe57897f-ea22-4f3e-a497-19c0649debc5-sys\") pod \"perf-node-gather-daemonset-kpc8h\" (UID: \"fe57897f-ea22-4f3e-a497-19c0649debc5\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-kpc8h"
Sep 16 12:48:48.917588 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:48.917397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fe57897f-ea22-4f3e-a497-19c0649debc5-podres\") pod \"perf-node-gather-daemonset-kpc8h\" (UID: \"fe57897f-ea22-4f3e-a497-19c0649debc5\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-kpc8h"
Sep 16 12:48:48.917588 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:48.917491    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe57897f-ea22-4f3e-a497-19c0649debc5-sys\") pod \"perf-node-gather-daemonset-kpc8h\" (UID: \"fe57897f-ea22-4f3e-a497-19c0649debc5\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-kpc8h"
Sep 16 12:48:48.917588 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:48.917549    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe57897f-ea22-4f3e-a497-19c0649debc5-lib-modules\") pod \"perf-node-gather-daemonset-kpc8h\" (UID: \"fe57897f-ea22-4f3e-a497-19c0649debc5\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-kpc8h"
Sep 16 12:48:48.917588 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:48.917581    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fe57897f-ea22-4f3e-a497-19c0649debc5-proc\") pod \"perf-node-gather-daemonset-kpc8h\" (UID: \"fe57897f-ea22-4f3e-a497-19c0649debc5\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-kpc8h"
Sep 16 12:48:48.917810 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:48.917583    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fe57897f-ea22-4f3e-a497-19c0649debc5-podres\") pod \"perf-node-gather-daemonset-kpc8h\" (UID: \"fe57897f-ea22-4f3e-a497-19c0649debc5\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-kpc8h"
Sep 16 12:48:48.917810 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:48.917632    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fe57897f-ea22-4f3e-a497-19c0649debc5-proc\") pod \"perf-node-gather-daemonset-kpc8h\" (UID: \"fe57897f-ea22-4f3e-a497-19c0649debc5\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-kpc8h"
Sep 16 12:48:48.917810 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:48.917648    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe57897f-ea22-4f3e-a497-19c0649debc5-lib-modules\") pod \"perf-node-gather-daemonset-kpc8h\" (UID: \"fe57897f-ea22-4f3e-a497-19c0649debc5\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-kpc8h"
Sep 16 12:48:48.917810 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:48.917713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btdnp\" (UniqueName: \"kubernetes.io/projected/fe57897f-ea22-4f3e-a497-19c0649debc5-kube-api-access-btdnp\") pod \"perf-node-gather-daemonset-kpc8h\" (UID: \"fe57897f-ea22-4f3e-a497-19c0649debc5\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-kpc8h"
Sep 16 12:48:48.928002 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:48.927980    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btdnp\" (UniqueName: \"kubernetes.io/projected/fe57897f-ea22-4f3e-a497-19c0649debc5-kube-api-access-btdnp\") pod \"perf-node-gather-daemonset-kpc8h\" (UID: \"fe57897f-ea22-4f3e-a497-19c0649debc5\") " pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-kpc8h"
Sep 16 12:48:49.060596 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:49.060515    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-kpc8h"
Sep 16 12:48:49.134133 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:49.134102    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-zpqj5_8bdb55fb-457e-4917-8168-53b1a0695c89/download-server/0.log"
Sep 16 12:48:49.336772 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:49.336747    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2psvv/perf-node-gather-daemonset-kpc8h"]
Sep 16 12:48:49.338215 ip-10-0-137-30 kubenswrapper[2579]: W0916 12:48:49.338188    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfe57897f_ea22_4f3e_a497_19c0649debc5.slice/crio-f5571f6d464e94880065b2b397851a9585fb27fa06ee0cc6e51d9d6489eceae2 WatchSource:0}: Error finding container f5571f6d464e94880065b2b397851a9585fb27fa06ee0cc6e51d9d6489eceae2: Status 404 returned error can't find the container with id f5571f6d464e94880065b2b397851a9585fb27fa06ee0cc6e51d9d6489eceae2
Sep 16 12:48:49.821532 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:49.821444    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-5b5wz_da760c1f-86c9-4430-a440-34b5dcf6a5c4/volume-data-source-validator/0.log"
Sep 16 12:48:50.005358 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:50.005322    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-kpc8h" event={"ID":"fe57897f-ea22-4f3e-a497-19c0649debc5","Type":"ContainerStarted","Data":"46a6b650701504efe117a58a83affd3d77cdd6746809a359fa559b53aca8453d"}
Sep 16 12:48:50.005577 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:50.005479    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-kpc8h"
Sep 16 12:48:50.005577 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:50.005504    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-kpc8h" event={"ID":"fe57897f-ea22-4f3e-a497-19c0649debc5","Type":"ContainerStarted","Data":"f5571f6d464e94880065b2b397851a9585fb27fa06ee0cc6e51d9d6489eceae2"}
Sep 16 12:48:50.033599 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:50.033545    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-kpc8h" podStartSLOduration=2.033530659 podStartE2EDuration="2.033530659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 12:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 12:48:50.031348901 +0000 UTC m=+2723.732420230" watchObservedRunningTime="2026-09-16 12:48:50.033530659 +0000 UTC m=+2723.734601984"
Sep 16 12:48:51.063056 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:51.063019    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mlc2p_096c53b4-6072-4b7e-9d0c-f2ffb8e13694/dns/0.log"
Sep 16 12:48:51.086548 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:51.086520    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mlc2p_096c53b4-6072-4b7e-9d0c-f2ffb8e13694/kube-rbac-proxy/0.log"
Sep 16 12:48:51.220843 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:51.220808    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q4hbf_b38a0789-177d-497c-8d7f-7d7177f5015e/dns-node-resolver/0.log"
Sep 16 12:48:51.762872 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:51.762847    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ldcrr_a6a91efe-a2f3-4829-9777-1ea089f08030/node-ca/0.log"
Sep 16 12:48:52.891364 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:52.891336    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp_54b8d903-4ad2-41f5-8db2-a22e9c8e76ca/istio-proxy/0.log"
Sep 16 12:48:53.382124 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:53.382095    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h6qjt_6ab958a3-431d-4406-953e-c5440bb50bb5/istio-proxy/0.log"
Sep 16 12:48:53.384860 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:53.384836    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h6qjt_6ab958a3-431d-4406-953e-c5440bb50bb5/istio-proxy/1.log"
Sep 16 12:48:54.360033 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:54.354311    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4hvw6_3b4372da-bcb1-4e89-8cd1-84642aa32972/serve-healthcheck-canary/0.log"
Sep 16 12:48:55.067254 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:55.067228    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sx7z6_85173723-783e-41a8-8b46-0459ca49e9e1/kube-rbac-proxy/0.log"
Sep 16 12:48:55.118045 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:55.118015    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sx7z6_85173723-783e-41a8-8b46-0459ca49e9e1/exporter/0.log"
Sep 16 12:48:55.149736 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:55.149711    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sx7z6_85173723-783e-41a8-8b46-0459ca49e9e1/extractor/0.log"
Sep 16 12:48:56.020853 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:56.020827    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2psvv/perf-node-gather-daemonset-kpc8h"
Sep 16 12:48:57.469577 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:57.469540    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-n5f8s_f7b110d0-4396-476b-a69b-91f00e2405cf/manager/0.log"
Sep 16 12:48:57.729876 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:57.729842    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-gwhkn_2b325748-891a-4d74-9bd1-ccbc177d05f2/manager/0.log"
Sep 16 12:48:57.803486 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:57.803463    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-njsx5_6aaabad9-9969-41bf-a8e3-620e6f1772f5/server/0.log"
Sep 16 12:48:57.919478 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:57.919447    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b5b57bf8b-2446x_c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4/manager/0.log"
Sep 16 12:48:57.952274 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:57.952249    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b5b57bf8b-2446x_c2e3d369-b66e-41f3-8f39-4d90cfd9fdc4/copy-manifests/0.log"
Sep 16 12:48:59.085063 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:59.085036    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp_54b8d903-4ad2-41f5-8db2-a22e9c8e76ca/istio-proxy/0.log"
Sep 16 12:48:59.560308 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:59.560217    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h6qjt_6ab958a3-431d-4406-953e-c5440bb50bb5/istio-proxy/0.log"
Sep 16 12:48:59.562563 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:59.562545    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h6qjt_6ab958a3-431d-4406-953e-c5440bb50bb5/istio-proxy/1.log"
Sep 16 12:48:59.575194 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:59.575170    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-74d55495b7-pcng2_13e5e417-b667-4c12-ab6c-d0c3aafb6eb0/manager/0.log"
Sep 16 12:48:59.601537 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:48:59.601513    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-wlxxw_c376e5cf-86e5-4bed-b75b-70c39ad18cac/openshift-lws-operator/0.log"
Sep 16 12:49:01.105393 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:49:01.105351    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-wrsnp_54b8d903-4ad2-41f5-8db2-a22e9c8e76ca/istio-proxy/0.log"
Sep 16 12:49:01.605512 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:49:01.605475    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h6qjt_6ab958a3-431d-4406-953e-c5440bb50bb5/istio-proxy/0.log"
Sep 16 12:49:01.608203 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:49:01.608179    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-h6qjt_6ab958a3-431d-4406-953e-c5440bb50bb5/istio-proxy/1.log"
Sep 16 12:49:04.716697 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:49:04.716651    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-6m4db_49ccefde-2d94-4218-adef-2f8d3a268331/migrator/0.log"
Sep 16 12:49:04.740339 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:49:04.740312    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-6m4db_49ccefde-2d94-4218-adef-2f8d3a268331/graceful-termination/0.log"
Sep 16 12:49:05.142077 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:49:05.142047    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-8f59b_f2dca0f8-99b0-4189-b1c0-329c54fdbc24/kube-storage-version-migrator-operator/1.log"
Sep 16 12:49:05.142952 ip-10-0-137-30 kubenswrapper[2579]: I0916 12:49:05.142934    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-8f59b_f2dca0f8-99b0-4189-b1c0-329c54fdbc24/kube-storage-version-migrator-operator/0.log"